Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus includes: a three-dimensional model structuring section configured to generate, when an image pickup signal related to a region in a subject is inputted from an image pickup apparatus configured to pick up an image of an inside of the subject, three-dimensional data representing a shape of the region based on the image pickup signal; and an image generation section configured to perform, on the three-dimensional data generated by the three-dimensional model structuring section, processing of allowing visual recognition of a boundary region between a structured region that is a region, an image of which is picked up by the image pickup apparatus, and an unstructured region that is a region, an image of which is yet to be picked up by the image pickup apparatus, and generate a three-dimensional image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of PCT/JP2016/078396filed on Sep. 27, 2016 and claims benefit of Japanese Application No.2015-190133 filed in Japan on Sep. 28, 2015, the entire contents ofwhich are incorporated herein by this reference.

BACKGROUND OF INVENTION 1. Field of the Invention

The present invention relates to an image processing apparatus and animage processing method that observe a subject by using an endoscope.

2. Description of the Related Art

Recently, an endoscope system using an endoscope has been widely used inmedical and industrial fields. For example, in the medical field, anendoscope needs to be inserted into an organ having a complicatedluminal shape in a subject to observe or examine an inside of the organin detail in some cases.

For example, a conventional example of Japanese Patent No. 5354494proposes an endoscope system that generates and displays a luminal shapeof the organ from an endoscope image picked up by an endoscope topresent a region observed by the endoscope.

SUMMARY OF THE INVENTION

An image processing apparatus according to an aspect of the presentinvention includes: a three-dimensional model structuring sectionconfigured to generate, when an image pickup signal related to a regionin a subject is inputted from an image pickup apparatus configured topick up an image of an inside of the subject, three-dimensional datarepresenting a shape of the region based on the image pickup signal; andan image generation section configured to perform, on thethree-dimensional data generated by the three-dimensional modelstructuring section, processing of allowing visual recognition of aboundary region between a structured region that is a region, an imageof which is picked up by the image pickup apparatus, and an unstructuredregion that is a region, an image of which is yet to be picked up by theimage pickup apparatus, and generate a three-dimensional image.

An image processing method according to an aspect of the presentinvention includes: generating, by a three-dimensional model structuringsection, when an image pickup signal related to a region in a subject isinputted from an image pickup apparatus configured to pick up an imageof an inside of the subject, three-dimensional data representing a shapeof the region based on the image pickup signal; and performing, by animage generation section, on the three-dimensional data generated by thethree-dimensional model structuring section, processing of allowingvisual recognition of a boundary region between a structured region thatis a region, an image of which is picked up by the image pickupapparatus, and an unstructured region that is a region, an image ofwhich is yet to be picked up by the image pickup apparatus, andgenerating a three-dimensional image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the entire configuration of anendoscope system according to a first embodiment of the presentinvention;

FIG. 2 is a diagram illustrating the configuration of an imageprocessing apparatus in the first embodiment;

FIG. 3A is an explanatory diagram illustrating renal pelvis and calyx ina state in which an insertion section of an endoscope is inserted;

FIG. 3B is a diagram illustrating an exemplary situation in which a 3Dmodel image displayed on a monitor in accordance with change of anobservation region along with an insertion operation of the endoscope isupdated;

FIG. 3C is a diagram illustrating an exemplary situation in which the 3Dmodel image displayed on the monitor in accordance with change of theobservation region along with the insertion operation of the endoscopeis updated;

FIG. 3D is a diagram illustrating an exemplary situation in which the 3Dmodel image displayed on the monitor in accordance with change of theobservation region along with the insertion operation of the endoscopeis updated;

FIG. 4 is a diagram illustrating a relation between a front surfacecorresponding to the order of apexes of a triangle as a polygon used tostructure a 3D model image, and a normal vector;

FIG. 5 is a flowchart illustrating processing of an image processingmethod according to the first embodiment;

FIG. 6 is a flowchart illustrating contents of processing according tothe first embodiment;

FIG. 7 is an explanatory diagram illustrating a situation in whichpolygons are set on a 3D-shaped surface;

FIG. 8 is a flowchart illustrating detail of processing of setting thenormal vector in FIG. 6 and determining an inner surface and an outersurface;

FIG. 9 is a diagram illustrating a polygon list produced when polygonsare set as illustrated in FIG. 7;

FIG. 10 is a diagram illustrating a polygon list generated by setting anormal vector for the polygon list illustrated in FIG. 9;

FIG. 11 is a diagram illustrating a situation in which normal vectorsare set to respective adjacent polygons set to draw an observed innersurface;

FIG. 12 is an explanatory diagram of an operation of determining thedirection of a normal vector by using position information of a positionsensor when the position sensor is provided at a distal end portion;

FIG. 13 is a diagram illustrating a 3D model image displayed on themonitor when enhanced display is not selected;

FIG. 14 is a diagram schematically illustrating the periphery of aboundary in a 3D model image;

FIG. 15 is a diagram illustrating a polygon list corresponding to thecase illustrated in FIG. 14;

FIG. 16 is a diagram illustrating a boundary list produced by extractionof boundary sides;

FIG. 17 is a diagram illustrating a 3D model image displayed on themonitor when enhanced display is selected;

FIG. 18 is a flowchart illustrating contents of processing in a firstmodification of the endoscope system according to the first embodiment;

FIG. 19 is an explanatory diagram for description of the operationillustrated in FIG. 18;

FIG. 20 is a diagram illustrating a 3D model image displayed on themonitor when enhanced display is selected in the first modification;

FIG. 21 is a flowchart illustrating contents of processing in a secondmodification of the endoscope system according to the first embodiment;

FIG. 22 is an explanatory diagram of processing in the secondmodification;

FIG. 23 is a diagram illustrating a 3D model image generated by thesecond modification and displayed on the monitor;

FIG. 24 is a flowchart illustrating contents of processing in a thirdmodification of the endoscope system according to the first embodiment;

FIG. 25 is an explanatory diagram of processing in the thirdmodification;

FIG. 26 is a diagram illustrating a 3D model image generated by thethird modification and displayed on the monitor;

FIG. 27 is a flowchart illustrating contents of processing in a fourthmodification of the endoscope system according to the first embodiment;

FIG. 28 is an explanatory diagram of processing in the fourthmodification;

FIG. 29 is a diagram illustrating a 3D model image generated by thefourth modification and displayed on the monitor;

FIG. 30A is a diagram illustrating the configuration of an imageprocessing apparatus in a fifth modification of the first embodiment;

FIG. 30B is a flowchart illustrating contents of processing in the fifthmodification of the endoscope system according to the first embodiment;

FIG. 31 is a diagram illustrating a 3D model image generated by thefifth modification and displayed on the monitor;

FIG. 32 is a flowchart illustrating contents of processing in a sixthmodification of the endoscope system according to the first embodiment;

FIG. 33 is a diagram illustrating a 3D model image generated by thesixth modification and displayed on the monitor;

FIG. 34 is a diagram illustrating the configuration of an imageprocessing apparatus in a seventh modification of the first embodiment;

FIG. 35 is a flowchart illustrating contents of processing in theseventh modification;

FIG. 36 is a diagram illustrating a 3D model image generated by theseventh modification and displayed on the monitor when enhanced displayand index display are selected;

FIG. 37 is a diagram illustrating a 3D model image generated by theseventh modification and displayed on the monitor when enhanced displayis not selected but index display is selected;

FIG. 38 is a flowchart illustrating contents of processing of generatingan index in an eighth modification of the first embodiment;

FIG. 39 is an explanatory diagram of FIG. 38;

FIG. 40 is an explanatory diagram of a modification of FIG. 38;

FIG. 41 is a diagram illustrating a 3D model image generated by theeighth modification and displayed on the monitor;

FIG. 42 is a diagram illustrating the configuration of an imageprocessing apparatus in a ninth modification of the first embodiment;

FIG. 43A is a diagram illustrating a 3D model image generated by theninth modification and displayed on the monitor;

FIG. 43B is a diagram illustrating a 3D model image before beingrotated;

FIG. 43C is a diagram illustrating a 3D model image before beingrotated;

FIG. 43D is an explanatory diagram when an unstructured region isdisplayed in an enlarged manner;

FIG. 44 is a diagram illustrating the configuration of an imageprocessing apparatus in a tenth modification of the first embodiment;

FIG. 45 is a diagram illustrating 3D shape data including a boundaryacross which a threshold is exceeded or not;

FIG. 46 is a diagram illustrating 3D shape data of a target ofdetermination by a determination section and the direction of an axis ofa primary component of the 3D shape data;

FIG. 47 is a diagram obtained by projecting the coordinates of aboundary illustrated in FIG. 46 onto a plane orthogonal to an axis of afirst primary component;

FIG. 48 is a diagram illustrating the configuration of an imageprocessing apparatus in an eleventh modification of the firstembodiment;

FIG. 49 is a flowchart illustrating contents of processing in theeleventh modification;

FIG. 50 is an explanatory diagram of processing in the eleventhmodification;

FIG. 51 is a diagram illustrating a core line image generated by theeleventh modification; and

FIG. 52 is a diagram illustrating the configuration of an imageprocessing apparatus in a twelfth modification of the first embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described below withreference to the accompanying drawings.

First Embodiment

An endoscope system 1 illustrated in FIG. 1 includes an endoscope 2Athat is inserted into a subject, a light source apparatus 3 configuredto supply illumination light to the endoscope 2A, a video processor 4 asa signal processing apparatus configured to perform signal processingfor an image pickup section provided to the endoscope 2A, a monitor 5 asan endoscope image display apparatus configured to display an endoscopeimage generated by the video processor 4, a UPD apparatus 6 as aninsertion section shape detection apparatus configured to detect aninsertion section shape of the endoscope 2A through a sensor provided inthe endoscope 2A, an image processing apparatus 7 configured to performimage processing of generating a three-dimensional (also abbreviated as3D) model image from a two-dimensional image, and a monitor 8 as adisplay apparatus configured to display the 3D model image generated bythe image processing apparatus 7. Note that an image processingapparatus 7A including the UPD apparatus 6 as illustrated with a dottedline may be used in place of the image processing apparatus 7 separatelyprovided from the UPD apparatus 6 illustrated with a solid line inFIG. 1. The UPD apparatus 6 may be omitted when position information isestimated from an image in the processing of generating athree-dimensional model image.

The endoscope 2A includes an insertion section 11 that is inserted into,for example, a ureter 10 as part of a predetermined luminal organ (alsosimply referred to as a luminal organ) that is a subject to be observedin a patient 9, an operation section 12 provided at a rear end (baseend) of the insertion section 11, and an universal cable 13 extendingfrom the operation section 12, and a light guide connector 14 providedat an end part of the universal cable 13 is detachably connected with alight guide connector reception of the light source apparatus 3.

Note that the ureter 10 communicates with a renal pelvis 51 a and arenal calyx 51 b on a deep part side (refer to FIG. 3A).

The insertion section 11 includes a distal end portion 15 provided at aleading end, a bendable bending portion 16 provided at a rear end of thedistal end portion 15, and a flexible pipe section 17 extending from arear end of the bending portion 16 to a front end of the operationsection 12.

The operation section 12 is provided with a bending operation knob 18for a bending operation of the bending portion 16.

As illustrated in a partially enlarged view in FIG. 1, a light guide 19that transmits illumination light is inserted in the insertion section11, and a leading end of the light guide 19 is attached to anillumination window of the distal end portion 15, whereas a rear end ofthe light guide 19 reaches the light guide connector 14.

Illumination light generated at a light source lamp 21 of the lightsource apparatus 3 is condensed through a light condensing lens 22 andincident on the light guide connector 14, and the light guide 19 emitstransmitted illumination light from a leading surface attached to theillumination window.

An optical image of an observation target site (also referred to as anobject) in the luminal organ illuminated with an illumination light isformed at an imaging position of an objective optical system 23 throughthe objective optical system 23 attached to an observation window (imagepickup window) provided adjacent to the illumination window of thedistal end portion 15. The image pickup plane of, for example, acharge-coupled device (abbreviated as CCD) 24 as an image pickup deviceis disposed at the imaging position of the objective optical system 23.The CCD 24 has a predetermined view angle.

The objective optical system 23 and the CCD 24 serve as an image pickupsection (or image pickup apparatus) 25 configured to pick up an image ofthe inside of the luminal organ. Note that the view angle of the CCD 24also depends on an optical property (for example, the focal length) ofthe objective optical system 23, and thus may be referred to as the viewangle of the image pickup section 25 with taken into consideration theoptical property of the objective optical system 23 or the view angle ofobservation using the objective optical system.

The CCD 24 is connected with one end of a signal line 26 inserted in,for example, the insertion section 11, and the other end of the signalline 26 extends to a signal connector 28 at an end part of theconnection cable 27 through a connection cable 27 (or a signal lineinside the connection cable 27) connected with the light guide connector14. The signal connector 28 is detachably connected with a signalconnector reception of the video processor 4.

The video processor 4 includes a driver 31 configured to generate a CCDdrive signal, and a signal processing circuit 32 configured to performsignal processing on an output signal from the CCD 24 to generate animage signal (video signal) to be displayed as an endoscope image on themonitor 5. The driver 31 applies the CCD drive signal to the CCD 24through, for example, the signal line 26, and upon the application ofthe CCD drive signal, the CCD 24 outputs, as an output signal, an imagepickup signal obtained through optical-electrical conversion of anoptical image formed on the image pickup plane.

Namely, the image pickup section 25 includes the objective opticalsystem 23 and the CCD 24 and is configured sequentially generate atwo-dimensional image pickup signal by receiving return light from aregion in a subject irradiated with illumination light from theinsertion section 11 and output the generated two-dimensional imagepickup signal.

The image pickup signal outputted from the CCD 24 is converted into animage signal by the signal processing circuit 32 and the signalprocessing circuit 32 outputs the image signal to the monitor 5 from anoutput end of the signal processing circuit 32. The monitor 5 displaysan image corresponding to an optical image formed on the image pickupplane of the CCD 24 and picked up at a predetermined view angle (in arange of view angle), as an endoscope image in an endoscope imagedisplay area (simply abbreviated as an image display area) 5 a. FIG. 1illustrates a situation in which, when the image pickup plane of the CCD24 is, for example, a square, an endoscope image substantially shaped inan octagon obtained by truncating the four corners of the square isdisplayed.

The endoscope 2A includes, for example, in the light guide connector 14,a memory 30 storing information unique to the endoscope 2A, and thememory 30 stores view angle data (or view angle information) asinformation indicating the view angle of the CCD 24 mounted on theendoscope 2A. When the light guide connector 14 is connected with thelight source apparatus 3, a reading circuit 29 a provided inside thelight source apparatus 3 reads view angle data through an electricalcontact connected with the memory 30.

The reading circuit 29 a outputs the read view angle data to the imageprocessing apparatus 7 through a communication line 29 b. The readingcircuit 29 a also outputs read data on the number of pixels of the CCD24 to the driver 31 and the signal processing circuit 32 of the videoprocessor 4 through a communication line 29 c. The driver 31 generates aCCD drive signal in accordance with the inputted data on the number ofpixels, and the signal processing circuit 32 performs signal processingcorresponding to the data on the number of pixels.

Note that the exemplary configuration in FIG. 1 illustrates the case inwhich the reading circuit 29 a configured to read unique information inthe memory 30 is provided to the light source apparatus 3, but thereading circuit 29 a may be provided to the video processor 4.

The signal processing circuit 32 serves as an input section configuredto input generated two-dimensional endoscope image data (also referredto as image data) as, for example, a digital image signal to the imageprocessing apparatus 7.

In the insertion section 11, a plurality of source coils 34 functioningas a sensor configured to detect the insertion shape of the insertionsection 11 being inserted into a subject are disposed at an appropriateinterval in a longitudinal direction of the insertion section 11. In thedistal end portion 15, two source coils 34 a and 34 b are disposed inthe longitudinal direction of the insertion section 11, and a sourcecoil 34 c is disposed in, for example, a direction orthogonal to a linesegment connecting the two source coils 34 a and 34 b. The direction ofthe line segment connecting the source coils 34 a and 34 b issubstantially aligned with an optical axis direction (or sight linedirection) of the objective optical system 23 included in the imagepickup section 25, and a plane including the three source coils 34 a, 34b, and 34 c is substantially aligned with an up-down direction of on theimage pickup plane of the CCD 24.

Thus, a source coil position detection circuit 39 to be described laterinside the UPD apparatus 6 can detect the three-dimensional position ofthe distal end portion 15 and a longitudinal direction of the distal endportion 15 by detecting the three-dimensional positions of the threesource coils 34 a, 34 b, and 34 c, and in other words, thethree-dimensional position of the objective optical system 23 includedin the image pickup section 25 and disposed at a known distance fromeach of the three source coils 34 a, 34 b, and 34 c and the sight linedirection (optical axis direction) of the objective optical system 23can be detected by detecting the three-dimensional positions of thethree source coils 34 a, 34 b, and 34 c at the distal end portion 15.

The source coil position detection circuit 39 serves as an informationacquisition section configured to acquire information on thethree-dimensional position and the sight line direction of the objectiveoptical system 23.

Note that the image pickup section 25 in the endoscope 2A illustrated inFIG. 1 has a configuration in which the image pickup plane of the CCD 24is disposed at the imaging position of the objective optical system 23,but the present invention is applicable to an endoscope including animage pickup section having a configuration in which an image guide thattransmits an optical image by the objective optical system 23 isprovided between the objective optical system 23 and the CCD.

The plurality of source coils 34 including the three source coils 34 a,34 b, and 34 c are each connected with one end of the corresponding oneof a plurality of signal lines 35, and the other ends of the pluralityof signal lines 35 are each connected with a cable 36 extending from thelight guide connector 14, and a signal connector 36 a at an end part ofthe cable 36 is detachably connected with a signal connector receptionof the UPD apparatus 6.

The UPD apparatus 6 includes a source coil drive circuit 37 configuredto drive the plurality of source coils 34 to generate analternating-current magnetic field around each source coil 34, a sensecoil unit 38 including a plurality of sense coils and configured todetect the three-dimensional position of each source coil by detecting amagnetic field generated by the respective source coils, the source coilposition detection circuit 39 configured to detect the three-dimensionalpositions of the respective source coils based on detection signals bythe plurality of sense coils, and an insertion section shape detectioncircuit 40 configured to detect the insertion shape of the insertionsection 11 based on the three-dimensional positions of the respectivesource coils detected by the source coil position detection circuit 39and generate an image in the insertion shape.

The three-dimensional position of each source coil is detected andmanaged in a coordinate system of the UPD apparatus 6.

As described above, the source coil position detection circuit 39 servesas an information acquisition section configured to acquire informationon the observation position (three-dimensional position) and the sightline direction of the objective optical system 23. In a more limitedsense, the source coil position detection circuit 39 and the threesource coils 34 a, 34 b, and 34 c serve as an information acquisitionsection configured to acquire information on the observation positionand the sight line direction of the objective optical system 23.

The endoscope system 1 (and the image processing apparatus 7) accordingto the present embodiment may employ an endoscope 2B illustrated with adouble-dotted and dashed line in FIG. 1 (in place of the endoscope 2A).

The endoscope 2B is provided with the insertion section 11 including nosource coils 34 in the endoscope 2A. In this endoscope, no source coils34 a, 34 b, and 34 c are disposed in the distal end portion 15 asillustrated in an enlarged view. When the endoscope 2B is connected withthe light source apparatus 3 and the video processor 4, the readingcircuit 29 a reads unique information in the memory 30 in the lightguide connector 14 and outputs the unique information to the imageprocessing apparatus 7. The image processing apparatus 7 recognizes thatthe endoscope 2B is an endoscope including no source coils.

The image processing apparatus 7 estimates the observation position andthe sight line direction of the objective optical system 23 by imageprocessing without using the UPD apparatus 6.

In the endoscope system 1 according to the present embodiment, althoughnot illustrated, the inside of renal pelvis and calyx may be examined byusing an endoscope (denoted by 2C) in which the source coils 34 a, 34 b,and 34 c that allow detection of the observation position and the sightline direction of the objective optical system 23 provided to the distalend portion 15 are provided in the distal end portion 15.

In this manner, in the present embodiment, identification informationprovided to the endoscope 2I (I=A, B, or C) is used to examine theinside of renal pelvis and calyx with any of the endoscope 2A (or 2C)including a position sensor and the endoscope 2B including no positionsensor and structure a 3D model image from two-dimensional image dataacquired through the examination as described later.

When the endoscope 2A is used, the insertion section shape detectioncircuit 40 includes a first output end from which an image signal of theinsertion shape of the endoscope 2A is outputted, and a second outputend from which data (also referred to as position and direction data) onthe observation position and the sight line direction of the objectiveoptical system 23 detected by the source coil position detection circuit39 is outputted. Then, the data on the observation position and thesight line direction is outputted from the second output end to theimage processing apparatus 7. Note that the data on the observationposition and the sight line direction outputted from the second outputend may be outputted from the source coil position detection circuit 39serving as an information acquisition section.

FIG. 2 illustrates the configuration of the image processing apparatus7. The image processing apparatus 7 includes a control section 41configured to perform operation control of the image processingapparatus 7, an image processing section 42 configured to generate (orstructure) 3D shape data (or 3D model data) and a 3D model image, and aninformation storage section 43 configured to store information such asimage data.

An image signal of the 3D model image generated by the image processingsection 42 is outputted to the monitor 8, and the monitor 8 displays the3D model image generated by the image processing section 42.

The control section 41 and the image processing section 42 are connectedwith an input apparatus 44 including, for example, a keyboard and amouse to allow a user such as an operator to perform, through a displaycolor setting section 44 a of the input apparatus 44, selection (orsetting) of a display color in which a 3D model image is displayed, andto perform, through an enhanced display selection section 44 b,selection of enhanced display of a boundary between a structured regionand an unstructured region in the 3D model image to facilitate visualrecognition. Note that, for example, any parameter for image processingcan be inputted to the image processing section 42 through the inputapparatus 44.

The control section 41 is configured by, for example, a centralprocessing unit (CPU) and functions as a processing control section 41 aconfigured to control an image processing operation of the imageprocessing section 42 in accordance with setting or selection from theinput apparatus 44.

Identification information unique to the endoscope 2I is inputted fromthe memory 30 to the control section 41, and the control section 41performs identification of the endoscope 2B including no position sensoror the endoscope 2A or 2C including a position sensor based on typeinformation of the endoscope 2I in the identification information.

Then, when the endoscope 2B including no position sensor is used, theimage processing section 42 is controlled to estimate the observationposition and the sight line direction of the image pickup section 25 orthe objective optical system 23 acquired by the UPD apparatus 6 when theendoscope 2A or 2C including a position sensor is used.

In such a case, the image processing section 42 functions as anobservation position and sight line direction estimation processingsection 42 d configured to perform processing of estimating theobservation position and the sight line direction (of the image pickupsection 25 or the objective optical system 23) of the endoscope 2B byusing, for example, a luminance value of two-dimensional endoscope imagedata as illustrated with a dotted line in FIG. 2. Data on theobservation position and the sight line direction estimated by theobservation position and sight line direction estimation processingsection 42 d is stored in a position and direction data storage section43 a provided in a storage region of the information storage section 43.Note that the position of the distal end portion 15 may be estimated inplace of the observation position of the image pickup section 25 or theobjective optical system 23.

The image processing section 42 includes a 3D shape data structuringsection 42 a including a CPU, a digital signal processor (DSP), and thelike and configured to generate (or structure) 3D shape data (or 3Dmodel data) from two-dimensional endoscope image data inputted from thevideo processor 4, and an image generation section 42 b configured togenerate, for the 3D shape data generated (or structured) by the 3Dshape data structuring section 42 a, a structured region of a 3D modelimage structured for a two-dimensional image region that is observed (oran image of which is picked up) by the image pickup section 25 of theendoscope and generate a 3D model image that allows (facilitates) visualrecognition of an unstructured region of the 3D model imagecorresponding to a two-dimensional image region unobserved by the imagepickup section 25 of the endoscope. In other words, the image generationsection 42 b generates (or structures) a 3D model image for displayingan unstructured region of the 3D model image in such a manner thatallows visual check. The 3D model image generated by the imagegeneration section 42 b is outputted to the monitor 8 as a displayapparatus and displayed on the monitor 8. The image generation section42 b functions as an output section configured to output a 3D modelimage (or image of 3D model data) to the display apparatus.

The image processing section 42 includes an image update processingsection 42 o configured to perform processing of updating, for example,3D shape data based on change of a region (two-dimensional regioncorresponding to a three-dimensional region) included in two-dimensionaldata along with an insertion operation. Note that FIG. 2 illustrates anexample in which the image update processing section 42 o is providedoutside of the image generation section 42 b, but the image updateprocessing section 42 o may be provided in the image generation section42 b. In other words, the image generation section 42 b may include theimage update processing section 42 o. The image update processingsection 42 o may be provided to an image processing apparatus in eachmodification to be described later (not illustrated).

Note that the image processing section 42 and, for example, the 3D shapedata structuring section 42 a and the image generation section 42 binside the image processing section 42 may be each configured by, inplace of a CPU and a DSP, a LSI (large-scale integration) FPGA (fieldprogrammable gate array) as hardware configured by a computer program ormay be configured by any other dedicated electronic circuit.

The image generation section 42 b includes a polygon processing section42 c configured to set, for 3D shape data generated (or structured) bythe 3D shape data structuring section 42 a, two-dimensional polygons(approximately) expressing each three-dimensional local region in the 3Dshape data and perform image processing on the set polygons. Note thatFIG. 2 illustrates an exemplary configuration in which the imagegeneration section 42 b includes the polygon processing section 42 cinside, but it can be effectively regarded that the polygon processingsection 42 c forms the image generation section 42 b.

As described above, when the endoscope 2B including no position sensoris used, the image processing section 42 includes the observationposition and sight line direction estimation processing section 42 dconfigured to estimate the observation position and the sight linedirection (of the image pickup section 25 or the objective opticalsystem 23) of the endoscope 2B.

The information storage section 43 is configured by, for example, aflash memory, a RAM, a USB memory, or a hard disk apparatus, andincludes a position and direction data storage section 43 a configuredto store view angle data acquired from the memory 30 of the endoscopeand store observation position and sight line direction data estimatedby the observation position and sight line direction estimationprocessing section 42 d or acquired from the UPD apparatus 6, an imagedata storage section 43 b configured to store, for example, 3D modelimage data of the image processing section 42, and a boundary datastorage section 43 c configured to store a structured region of astructured 3D model image and boundary data as a boundary of thestructured region.

As illustrated in FIG. 3A, the insertion section 11 of the endoscope 2Iis inserted into the ureter 10 having a three-dimensional luminal shapeto examine renal pelvis and calyx 51 farther on the deep part side. Inthis case, the image pickup section 25 disposed at the distal endportion 15 of the insertion section 11 picks up an image of a region inthe view angle of the image pickup section 25, and the signal processingcircuit 32 generates a two-dimensional image by performing signalprocessing on image pickup signals sequentially inputted from the imagepickup section 25.

Note that the renal pelvis 51 a is indicated as a region illustratedwith a dotted line in FIG. 3A in the renal pelvis and calyx 51 on thedeep part side of the ureter 10, and the renal calyx 51 b is located onthe deep part side of the renal pelvis 51 a.

The 3D shape data structuring section 42 a to which two-dimensionalimage data is inputted generates 3D shape data corresponding totwo-dimensional image data picked up (observed) by the image pickupsection 25 of the endoscope 2I, by using observation position and sightline direction data acquired by the UPD apparatus 6 or observationposition and sight line direction data estimated by the observationposition and sight line direction estimation processing section 42 d.

In this case, the 3D shape data structuring section 42 a may estimate a3D shape from a corresponding single two-dimensional image by a methoddisclosed in, for example, the publication of Japanese Patent No.5354494 or a publicly known shape-from-shading method other than thispublication. In addition, a stereo method, a three-dimensional shapeestimation method by single-lens moving image pickup, a SLAM method, anda method of estimating a 3D shape in cooperation with a position sensor,which use two images or more are applicable. When a 3D shape isestimated, 3D shape data may be structured with reference to 3D imagedata acquired from a cross-sectional image acquisition apparatus such asa CT apparatus externally provided.

The following describes a specific method when the image processingsection 42 generates 3D model data in accordance with change of(two-dimensional data of) an observation region along with an insertionoperation of the endoscope 2I.

The 3D shape data structuring section 42 a generates 3D shape data fromany region included in a two-dimensional image pickup signal of asubject outputted from the image pickup section 25.

The image update processing section 42 o performs processing of updatinga 3D model image generated by the 3D shape data structuring section 42a, based on change of two-dimensional data along with the insertionoperation of the endoscope 2I.

More specifically, for example, when a first two-dimensional imagepickup signal generated at the image pickup section 25 upon reception ofreturn light from a first region in the subject is inputted, the 3Dshape data structuring section 42 a generates first 3D shape datacorresponding to the first region included in the first two-dimensionalimage pickup signal. The image update processing section 42 o stores thefirst 3D shape data generated by the 3D shape data structuring section42 a in the image data storage section 43 b.

When a second two-dimensional image pickup signal generated at the imagepickup section 25 upon reception of return light from a second regiondifferent from the first region is inputted after the first 3D shapedata is stored in the image data storage section, the 3D shape datastructuring section 42 a generates second 3D shape data corresponding tothe second region included in the second two-dimensional image pickupsignal. The image update processing section 42 o stores, in addition tothe first 3D shape data, the second 3D shape data generated by the 3Dshape data structuring section 42 a in the image data storage section 43b.

Then, the image update processing section 42 o generates a current 3Dmodel image by synthesizing the first 3D shape data and the second 3Dshape data stored in the image data storage section 43 b, and outputsthe generated 3D model image to the monitor 8.

Thus, when the distal end portion 15 of the endoscope 2I is moved by theinsertion operation, a 3D model image corresponding to any regionincluded in an endoscope image observed in the past from start of the 3Dmodel image generation to the current observation state of the distalend portion 15 is displayed on the monitor 8. The display region of the3D model image displayed on the monitor 8 increases with time elapse.

Note that, when a 3D model image is displayed on the monitor 8 by usingthe image update processing section 42 o, a (second) 3D model imagecorresponding only to a structured region that is already observed canbe displayed, but convenience can be improved for the user by displayinginstead a (first) 3D model image that allows visual recognition of aregion yet to be structured. Thus, the following description will bemainly made on an example in which the (first) 3D model image thatallows visual recognition of an unstructured region is displayed.

The image update processing section 42 o updates the (first) 3D modelimage based on change of a region included in endoscope image data asinputted two-dimensional data. The image update processing section 42 ocompares inputted current endoscope image data with endoscope image dataused to generate the (first) 3D model image right before the currentendoscope image data.

Then, when a detected change amount is equal to or larger than athreshold set as a comparison result in advance, the image updateprocessing section 42 o updates the past (first) 3D model image with the(first) 3D model image based on the current endoscope image data.

Note that, when updating the (first) 3D model image, the image updateprocessing section 42 o may use, for example, information on a leadingend position of the endoscope 2I, which changes along with the insertionoperation of the endoscope 2I. To achieve such processing, for example,the image processing apparatus 7 may be provided with a positioninformation acquisition section 81 as illustrated with a dotted line inFIG. 2.

The position information acquisition section 81 acquires leading endposition information as information indicating the leading end positionof the distal end portion 15 of the insertion section 11 of theendoscope 2I, and outputs the acquired leading end position informationto the image update processing section 42 o.

The image update processing section 42 o determines whether the leadingend position in accordance with the leading end position informationinputted from the position information acquisition section 81 haschanged from a past position. Then, when having acquired a determinationresult that the leading end position in accordance with the leading endposition information inputted from the position information acquisitionsection 81 has changed from the past position, the image updateprocessing section 42 o generates the current (first) 3D model imageincluding a (first) 3D model image part based on two-dimensional datainputted at a timing at which the determination result is acquired.Namely, the image update processing section 42 o updates the (first) 3Dmodel image before the change with a (new first) 3D model image (afterthe change).

The respective barycenters of the (first) 3D model image and the past(first) 3D model image may be calculated, and the update may beperformed when a detected change amount is equal to or larger than athreshold set as a comparison result in advance.

Alternatively, information used by the image update processing section42 o when updating the (first) 3D model image may be selected from amongtwo-dimensional data, a leading end position, and a barycenter inaccordance with, for example, an operation of the input apparatus 44 bythe user, or all of the two-dimensional data, the leading end position,and the barycenter may be selected. That is, the input apparatus 44functions as a selection section configured to allow selection of atleast one of two pieces (or two kinds) of information used by the imageupdate processing section 42 o when updating the (first) 3D model image.

The present endoscope system includes the endoscope 2I configured toobserve inside of a subject having a three-dimensional shape, the signalprocessing circuit 32 of the video processor 4 serving as an inputsection configured to input two-dimensional data of (the inside of) thesubject observed by the endoscope 2I, the 3D shape data structuringsection 42 a or the image generation section 42 b serving as athree-dimensional model image generation section configured to generatea three-dimensional model image that represents the shape of the subjectand is to be outputted to the monitor 8 as a display section based on aregion included in the two-dimensional data of the subject inputted bythe input section, and the image update processing section 42 oconfigured to update the three-dimensional model image to be outputtedto the display section based on change of the region included in thetwo-dimensional data along with an insertion operation of the endoscope2I and output the updated three-dimensional model image to the displaysection.

Besides processing of storing the first 3D shape data and the second 3Dshape data in the image data storage section 43 b, generating a 3D modelimage, and outputting the generated 3D model image to the monitor 8, theimage update processing section 42 o may also be configured to output a3D model image generated by performing any processing other than theprocessing to the monitor 8.

More specifically, the image update processing section 42 o may perform,for example, processing of storing only the first 3D shape data in theimage data storage section 43 b, generating a 3D model image bysynthesizing the first 3D shape data read from the image data storagesection 43 b and the second 3D shape data inputted after the first 3Dshape data is stored in the image data storage section 43 b, andoutputting the generated 3D model image to the monitor 8. Alternatively,the image update processing section 42 o may perform, for example,processing of generating a 3D model image by synthesizing the first 3Dshape data and the second 3D shape data without storing the first 3Dshape data and the second 3D shape data in the image data storagesection 43 b, storing the 3D model image in the image data storagesection 43 b, and outputting the 3D model image read from the image datastorage section 43 b to the monitor 8.

Alternatively, the image update processing section 42 o is not limitedto storage of 3D shape data generated by the 3D shape data structuringsection 42 a in the image data storage section 43 b, but may store, inthe image data storage section 43 b, a two-dimensional image pickupsignal generated at the image pickup section 25 when return light fromthe inside of the subject is received.

More specifically, for example, when the first two-dimensional imagepickup signal generated at the image pickup section 25 upon reception ofreturn light from the first region in the subject is inputted, the imageupdate processing section 42 o stores the first two-dimensional imagepickup signal in the image data storage section 43 b.

When the second two-dimensional image pickup signal generated at theimage pickup section 25 upon reception of return light from the secondregion different from the first region is inputted after the firsttwo-dimensional image pickup signal is stored in the image data storagesection 43 b, the image update processing section 42 o stores, inaddition to the first two-dimensional image pickup signal, the secondtwo-dimensional image pickup signal in the image data storage section 43b.

Then, the image update processing section 42 o generates athree-dimensional model image corresponding to the first region and thesecond region based on the first image pickup signal and the secondimage pickup signal stored in the image data storage section 43 b, andoutputs the three-dimensional model image to the monitor 8.

The following describes a display timing that is a timing at which theimage update processing section 42 o outputs the three-dimensional modelimage corresponding to the first region and the second region to themonitor 8.

For example, at each predetermined duration (for example, every second),the image update processing section 42 o updates 3D shape data stored inthe image data storage section 43 b and outputs the updated 3D shapedata to the monitor 8. Then, according to such processing by the imageupdate processing section 42 o, a three-dimensional model imagecorresponding to a two-dimensional image pickup signal of the inside ofan object sequentially inputted to the image processing apparatus 7 canbe displayed on the monitor 8 while being updated.

Note that, for example, when a trigger signal as a trigger for updatingan image is inputted in response to an operation of the input apparatus44 by the user, the image update processing section 42 o may update 3Dshape data stored in the image data storage section 43 b at eachpredetermined duration (for example, every second), generate athree-dimensional model image in accordance with the 3D shape data, andoutput the three-dimensional model image to the monitor 8. According tosuch processing by the image update processing section 42 o, thethree-dimensional model image can be displayed on the monitor 8 whilebeing updated at a desired timing, and thus convenience can be improvedfor the user.

For example, when having sensed that no treatment instrument such as abasket is present in an endoscope image corresponding to atwo-dimensional image pickup signal generated by the image pickupsection 25 (namely, when having sensed that the endoscope is inserted ina pipe line, not in treatment of a lesion site), the image updateprocessing section 42 o may output the three-dimensional model image tothe monitor 8 while updating the three-dimensional model image.

According to the processing as described above, for example, a 3D modelimage displayed (in a display region adjacent to an endoscope image) onthe monitor 8 is updated in the following order of I3 oa in FIG. 3B, I3ob in FIG. 3C, and I3 oc in FIG. 3D in response to change of(two-dimensional data of) the observation region along with an insertionoperation of the endoscope 2I inserted into renal pelvis and calyx.

The 3D model image I3 oa illustrated in FIG. 3B is an image generatedbased on an endoscope image observed up to an insertion positionillustrated on the right side in FIG. 3B. An upper end part in the 3Dmodel image I3 oa is a boundary Ba between a structured regioncorresponding to an observation region that is observed and anunobserved region, and the boundary Ba portion is displayed in a colordifferent from the color of the structured region.

Note that an arrow in the 3D model image I3 oa illustrated in FIG. 3Bindicates the position and the direction of the distal end portion 15 ofthe endoscope 2A (same in FIGS. 3C and 3D). The above-described arrow asan index indicating the position and the direction of the distal endportion 15 of the endoscope 2A may be superimposed in the 3D model imageI3 oa.

The 3D model image I3 ob illustrated in FIG. 3C is a 3D model imageupdated by adding a structured region to an unstructured region part inthe 3D model image I3 oa illustrated in FIG. 3B.

In the 3D model image I3 ob illustrated in FIG. 3C, boundaries Bb, Bc,and Bd with a plurality of unstructured regions are generated due tobifurcation parts halfway through the insertion. Note that the boundaryBd includes a part not attributable to a bifurcation part.

The 3D model image I3 oc illustrated in FIG. 3D is a 3D model imageupdated by adding a structured region to an unstructured region on anupper part side in the 3D model image I3 ob illustrated in FIG. 3C.

In the present embodiment, the insertion section 11 of the endoscope 2Iis inserted through the ureter 10 having a luminal shape into the renalpelvis and calyx 51 having a luminal shape on the deep part side of theureter 10. In this case, the 3D shape data structuring section 42 astructures hollow 3D shape data when the inner surface of the organhaving a luminal shape is observed.

The image generation section 42 b (the polygon processing section 42 c)sets polygons to the 3D shape data structured by the 3D shape datastructuring section 42 a and generates a 3D model image using thepolygons. In the present embodiment, the 3D model image is generated byperforming processing of bonding triangles as polygons onto the surfaceof the 3D shape data. That is, the 3D model image employs triangularpolygons as illustrated in FIG. 4. Typically, triangles or rectanglesare often used as polygons, but in the present embodiment, triangularpolygons are used. Note that the 3D shape data structuring section 42 amay directly generate (or structure) a 3D model image instead of the 3Dshape data.

Each polygon can be disassembled into a plane, sides, and apexes, andeach apex is described with 3D coordinates. The plane has front and backsurfaces, and one perpendicular normal vector is set to the plane.

The front surface of the plane is set by the order of description of theapexes of the polygon. For example, as illustrated in FIG. 4, the frontand back surface (of the plane) when described in the following order ofthree apexes v1, v2, and v3 correspond to the direction of a normalvector vn.

As described later, the setting of a normal vector corresponds todetermination of the front and back surfaces of a polygon to which thenormal vector is set, in other words, determination of whether eachpolygon on a 3D model image (indicating an observed region) formed byusing the polygons corresponds to the inner surface (or inner wall) orthe outer surface (or outer wall) of the luminal organ. In the presentembodiment, it is a main objective to observe or examine the innersurface of the luminal organ, and thus the following description will bemade on an example in which the inner surface of the luminal organ isassociated with the front surface of the plane of each polygon (theouter surface of the luminal organ is associated with the back surfaceof the plane of the polygon). When the inner and outer surfaces of aluminal structural body in a subject having a more complicated shape andincluding the luminal structural body inside are examined, the presentembodiment is also applicable to the complicated subject to distinguish(determine) the inner and outer surfaces.

Note that, as described later with reference to FIG. 6, each time whenthe insertion position of the insertion section 11 moves and a region ofa two-dimensional image acquired through observation by the image pickupsection 25 changes, the image processing section 42 repeats processingof generating 3D shape data of the changed region, updating 3D shapedata before the change with the generated 3D shape data, newly setting apolygon on the updated region appropriately by using the normal vector,and generating a 3D model image through addition (update).

The image generation section 42 b functions as an inner and outersurface determination section 42 e configured to determine, when addinga polygon, whether an observed local region represented by the plane ofthe polygon corresponds to the inner surface (inner wall) or the outersurface (outer wall) by using the normal vector of the polygon.

When enhanced display in which a boundary is displayed in an enhancedmanner is selected through the enhanced display selection section 44 bof the input apparatus 44, the image generation section 42 b functionsas a boundary enhancement processing section 42 f configured to display,in an enhanced manner, a boundary region with a structured region (as anobserved and structured region) (the boundary region also serves as aboundary with an unstructured region as a region yet to be observed andstructured) in the 3D model image. The boundary enhancement processingsection 42 f does not perform the processing of enhancing a boundaryregion (boundary part) when the enhanced display is not selected throughthe enhanced display selection section 44 b by the user.

In this manner, when a 3D model image is displayed on the monitor 8, theuser can select the enhanced display of a boundary with an unstructuredregion to facilitate visual recognition or select display of the 3Dmodel image on the monitor 8 without selecting the enhanced display.

The image generation section 42 b includes a (polygon) coloringprocessing section 42 g configured to color, in different colors, theinner and outer surfaces of the plane of a structured (in other words,observed) polygon with which a 3D model image is formed, in accordancewith a determination result of inner and outer surfaces. Note thatdifferent textures may be attached to a polygon instead of the coloringin different colors. The following description will be made on anexample in which the display color setting section 44 a is set to coloran inner surface (observed) in gray and an outer surface (unobserved) inwhite. Gray may be set to be close to white. The present embodiment isnot limited to the example in which the inner surface is colored in grayand the outer surface is colored in white (the coloring is performed bythe coloring processing section 42 g corresponding to a color set by thedisplay color setting section 44 a).

Note that, in the present embodiment, in a normal observation mode inwhich the inner surface of the luminal organ is an observation target,an unobserved region is the inner surface of the luminal organ, an imageof which is yet to be picked up by the image pickup section 25.

Then, when the unobserved region is displayed on a 3D model image toallow visual recognition by the operator, for example, duringobservation and examination with the endoscope 2I, any unstructuredregion existing on the 3D model image and corresponding to theunobserved region can be displayed in an image that allows easy visualrecognition in a 3D space by displaying the 3D model image in a shapeclose to the shape of the renal pelvis and calyx 51 illustrated in FIG.3A.

Thus, in the present embodiment, the image processing section 42generates, by using polygons, a 3D model image of the renal pelvis andcalyx 51 as a luminal organ illustrated in FIG. 3A when viewed from aviewpoint vertically above the sheet of the FIG. 3A in a predetermineddirection.

When the viewpoint is set outside of the luminal organ in this manner,it is difficult to display an actually observed region existing on theinner surface of a lumen in a manner that allows easy visual recognitionas an observed structured region on a 3D model image viewed from aviewpoint set on the outer surface of the lumen.

The difficulty can be avoided as described in the following methods (a),(b), and (c). The methods (a) and (b) are applicable to a double (ormultiplex) tubal structure, and the method (c) is applicable to a singletubal structure such as a renal pelvis.

(a) When a (drawn) 3D model image is viewed from a viewpoint, a regionof the outer surface covering an observed structured region on the 3Dmodel image is colored in a display color (for example, green) differentfrom gray as the color of the inner surface and white as the color ofthe outer surface. (b) As illustrated with a double-dotted and dashedline in FIG. 3A, for example, an illumination light source Ls is set ata viewpoint at a position vertically above the sheet of FIG. 3A, and anouter surface region covering a structured region on a 3D model imageobserved with illumination light radially emitted from the light sourceLs may be displayed in a display color (for example, green) colored inthe color of the illumination light of the illumination light source Ls.

(c) In a limited case in which only the inner surface of the luminalorgan is an observation target, the outer surface of the luminal organis not an observation target, and thus when the outer surface covers theobserved inner surface of the luminal organ, the outer surface may bedisplayed in a display color different from gray as the color of theinner surface. In such a case, white may be set as a display color inwhich the observed inner surface covered by the outer surface isdisplayed. In the following, a display color different (or easilydistinguishable) at least from gray (as a color in which the innersurface observed and not covered by the outer surface is displayed in adirect manner (in an exposing manner)) is used as a display color inwhich the outer surface when covering the observed inner surface of theluminal organ is displayed. In the present specification, the outersurface covering the observed inner surface is displayed in this mannerin a display color different from color (for example, gray) when theobserved inner surface is observed directly in an exposed state.

In the present embodiment, a background part of a 3D model image is setto have a background color (for example, blue) different from a color(gray) in which the observed inner surface is displayed in display ofthe 3D model image and the display color (for example, green) of theouter surface when the observed inner surface is covered by the outersurface in a double tubal structure, thereby achieving easy visualrecognition (display) of a boundary region as a boundary between astructured region and an unstructured region together with an observedstructured region. When the enhanced display is selected, the coloringprocessing section 42 g colors the boundary region in a color (forexample, red) different from gray, the display color, and the backgroundcolor for easier visual recognition.

Note that, in FIG. 1, the image processing apparatus 7 is providedseparately from the video processor 4 and the light source apparatus 3included in the endoscope apparatus, but the image processing apparatus7 may be provided in the same housing of the video processor 4 and thelight source apparatus 3.

The endoscope system 1 according to the present embodiment includes theendoscope 2I configured to observe the inside of the ureter 10 or therenal pelvis and calyx 51 as a subject having a three-dimensional shape,the signal processing circuit 32 of the video processor 4 serving as aninput section configured to input two-dimensional data of (the insideof) the subject observed by the endoscope 2I, the 3D shape datastructuring section 42 a serving as a three-dimensional modelstructuring section configured to generate (or structure)three-dimensional model data or three-dimensional shape data of thesubject based on the two-dimensional data of the subject inputted by theinput section, and the image generation section 42 b configured togenerate a three-dimensional model image that allows visual recognitionof an unstructured region (in other words, that facilitates visualrecognition of the unstructured region or in which the unstructuredregion can be visually recognized) as an unobserved region in thesubject based on the three-dimensional model data of a structuredregion, which is structured by the three-dimensional model structuringsection.

As illustrated in FIG. 5, an image processing method in the presentembodiment includes: an input step S1 at which the signal processingcircuit 32 of the video processor 4 inputs, to the image processingapparatus 7, two-dimensional image data as two-dimensional data of (theinside of) a subject observed by the endoscope 2I configured to observethe inside of the ureter 10 or the renal pelvis and calyx 51 as asubject having a three-dimensional shape; a three-dimensional modelstructuring step S2 at which the 3D shape data structuring section 42 agenerates (or structures) three-dimensional model data (3D shape data)of the subject based on the two-dimensional data (2D data) of thesubject inputted at the input step S1; and an image generation step S3at which the image generation section 42 b generates, based on thethree-dimensional model data of a structured region structured at thethree-dimensional model structuring step S2, a three-dimensional modelimage that allows visual recognition of an unstructured region (in otherwords, that facilitates visual recognition of the unstructured region orin which the unstructured region can be visually recognized) as anunobserved region in the subject. Note that contents of processingillustrated in FIG. 5 outline contents of processing illustrated in FIG.6 to be described below.

The following describes an operation according to the present embodimentwith reference to FIG. 6. FIG. 6 illustrates the procedure of mainprocessing by the endoscope system 1 according to the presentembodiment. Note that, in the processing illustrated in FIG. 6,different system configurations and image processing methods may beemployed between a case in which the enhanced display is not selectedand a case in which the enhanced display is selected.

As illustrated in FIG. 1, the operator connects the image processingapparatus 7 to the light source apparatus 3 and the video processor 4and connects the endoscope 2A, 2B, or 2C to the light source apparatus 3and the video processor 4 before performing an endoscope examination. Inthe examination, the insertion section 11 of the endoscope 2I isinserted into the ureter 10 of the patient 9. Then, as described at stepS11 in FIG. 6, the insertion section 11 of the endoscope 2I is insertedinto the renal pelvis and calyx 51 on the deep part side through theureter 10 as illustrated in FIG. 3A.

The image pickup section 25 is provided at the distal end portion 15 ofthe insertion section 11 and inputs an image pickup signal picked up(observed) in the view angle of the image pickup section 25 to thesignal processing circuit 32 of the video processor 4.

As described at step S12, the signal processing circuit 32 performssignal processing on the image pickup signal picked up by the imagepickup section 25 to generate (acquire) a two-dimensional image observedby the image pickup section 25. The signal processing circuit 32 inputs(two-dimensional image data obtained through A/D conversion of) thegenerated two-dimensional image to the image processing section 42 ofthe image processing apparatus 7.

As described at step S13, the 3D shape data structuring section 42 a ofthe image processing section 42 generates 3D shape data from theinputted two-dimensional image data by using information of a positionsensor when the endoscope 2A (or 2C) including the position sensor isused or by performing image processing to estimate a 3D shapecorresponding to an image region observed (by the image pickup section25) and estimating 3D shape data as 3D model data when the endoscope 2Bincluding no position sensor is used.

The 3D shape data may be generated from the two-dimensional image databy the method described above.

At the next step S14, the image generation section 42 b generates a 3Dmodel image by using polygons. As illustrated in FIG. 6, similarprocessing is repeated in a loop. Thus, at the second repetition orlater, the processing at step S14 continues the processing of generatinga 3D model image by using polygons at the last repetition (generating a3D model image for any new polygon and updating the previous 3D modelimage).

At the next step S15, the polygon processing section 42 c generatespolygons by a well-known method such as the method of marching cubesbased on the 3D shape data generated at step S13. FIG. 7 illustrates asituation in which polygons are generated based on the 3D shape datagenerated at step S13.

In 3D shape data (an outline shape part in FIG. 7) I3 a generated toillustrate a lumen, polygons are set onto the outer surface of the lumenwhen the lumen is viewed from a side, thereby generating a 3D modelimage I3 b.

Note that a 3D model image I3 c is then generated through coloringprocessing and displayed on the monitor 8. FIG. 7 illustrates polygonspO1, pO2, pO3, pO4, and the like.

At the next step S16, the polygon processing section 42 c sets a normalvector to each polygon set at the previous step S15 (to determinewhether an observed region is an inner surface).

At the next step S17, the inner and outer surface determination section42 e of the image generation section 42 b determines whether theobserved region is an inner surface by using the normal vector.Processing at steps S16 and S17 will be described later with referenceto FIG. 8.

At the next step S18, the coloring processing section 42 g of the imagegeneration section 42 b colors the plane of each polygon representingthe observed region (in gray for the inner surface or white for theouter surface) in accordance with a determination result at the previousstep S17.

At the next step S19, the control section 41 (or the boundaryenhancement processing section of the image generation section 42 b)determines whether the enhanced display is selected. When the enhanceddisplay is not selected, the process proceeds to processing at the nextstep S20. The next step S20 is followed by processing at steps S21 andS22.

When the enhanced display is selected, the process performs processingat steps S23, S24, and S25, and then proceeds to the processing at stepS20.

At step S20, the coloring processing section 42 g of the imagegeneration section 42 b colors an observed surface of a polygon in astructured region of the 3D model image when viewed (at a position setoutside of or separately from the 3D model image) in a predetermineddirection is an inner surface, in a color corresponding to a case inwhich the plane is hidden behind the outer surface.

Similarly to the double tubal structure described above, when anobserved surface of a polygon in a structured region of the 3D modelimage viewed in a predetermined direction is an inner surface and a 3Dmodel image in which the inner surface is covered by the outer surfaceis displayed, the outer surface is colored in a display color (forexample, green) different from gray as a display color indicating anobserved inner surface, white as the color of an observed outer surface,and the background color. Note that, when the 3D model image isdisplayed, an observed inner surface being exposed remains in gray,which is provided in the coloring processing at step S18.

At step S21 following the processing at step S20, the image processingsection 42 or the image generation section 42 b outputs an image signalof the 3D model image generated (by the above-described processing) tothe monitor 8, and the monitor 8 displays the generated 3D model image.

At the next step S22, the control section 41 determines whether theoperator inputs an instruction to end the examination through, forexample, the input apparatus 44.

When the instruction to end the examination is not inputted, the processreturns to the processing at step S11 or step S12 and repeats theabove-described processing. That is, when the insertion section 11 ismoved in the renal pelvis and calyx 51, the processing of generating 3Dshape data corresponding to a region newly observed by the image pickupsection 25 after the movement and generating a 3D model image for the 3Dshape data is repeated.

When the instruction to end the examination is inputted, the imageprocessing section 42 ends the processing of generating a 3D model imageas described at step S26, which ends the processing illustrated in FIG.6.

FIG. 13 illustrates the 3D model image I3 c displayed on the monitor 8halfway through the repetition of the above-described processing, forexample, after the processing at step S21 when the enhanced display isnot selected (when the processing at steps S23, S24, and S25 are notperformed).

The processing at steps S16 and S17 in FIG. 6 will be described nextwith reference to FIG. 8. Through the processing at step S15, asillustrated in FIG. 7, a plurality of polygons pO1, pO2, pO3, pO4, andthe like are set to the 3D shape data 13 a of the observed region. Thesepolygons pj (j=01, 02, 03, . . . ) are stored (held) as a polygon listin a table format illustrated in FIG. 9 in the information storagesection 43. The three apexes v1, v2, and v3 of each polygon pj are eachdetermined with a three-dimensional position vector value XXXX. Notethat the polygon list indicates the configuration of each polygon.

At the first step S31 in FIG. 8, the polygon processing section 42 cselects a polygon. As illustrated in FIG. 9, the polygon pO2 adjacent tothe polygon pO1 to which a normal vector indicated with XXXX is set isselected. Note that, a normal vector vn1 of the polygon pO1 is set inthe direction of a front surface indicating an observed inner surface asdescribed with reference to FIG. 4.

At the next step S32, the polygon processing section 42 c calculates anormal vector vn2 of the polygon pO2 as vn2=(v2−v1)×(v3−v1). Note that,to simplify description, the three-dimensional positions of the apexesv1, v2, and v3 are represented by using v1, v2, and v3, and, forexample, v2−v1 represents a vector extending from the three-dimensionalposition v1 to the three-dimensional position v2.

At the next step S33, the polygon processing section 42 c determineswhether the direction (or polarity) of the normal vector vn2 of thepolygon pO2 is same as the registered direction of the normal vector vn1of the polygon pO1.

To perform the determination, the polygon processing section 42 ccalculates the inner product of the normal vector vn1 of the polygon pO1adjacent to the polygon pO2 at an angle equal to or larger than 90degrees and the normal vector vn2 of the polygon pO2, and determinesthat the directions are same when the value of the inner product isequal to or larger than zero, or determines that the directions areinverted with respect to each other when the value is less than zero.

When it is determined that the directions are inverted with respect toeach other at step S33, the polygon processing section 42 c corrects thedirection of the normal vector vn2 at the next step S35. For example,the normal vector vn2 is corrected by multiplication by −1 andregistered, and the position vectors v2 and v3 in the polygon list areswapped.

After step S34 or when it is determined that the directions are same atstep S33, the polygon processing section 42 c determines whether allpolygons have normal vectors (normal vectors are set to all polygons) atstep S35.

The process returns to the processing at the first step S31 when thereis any polygon having no normal vector, or the processing illustrated inFIG. 8 is ended when all polygons have normal vectors. FIG. 10illustrates a polygon list obtained by setting normal vectors to thepolygon list illustrated in FIG. 9. FIG. 11 illustrates a situation inwhich, for example, the normal vector vn2 is set to the polygon pO2adjacent to the polygon pO1 through the processing illustrated in FIG.8. Note that, in FIG. 11, the upper part sides of the polygons 02 to 04correspond to the inner surface of the luminal organ (and the lowersides correspond to the outer surface).

In the above description, whether the directions of normal vectors aresame is determined by using an inner product in the determinationprocessing at step S33 in FIG. 8. This method is also applicable to theendoscope 2B including no position sensor.

However, when the endoscope 2A (or 2C) including a position sensor atthe distal end portion 15 is used, information of the position sensor asillustrated in FIG. 12 may be used to determine whether the direction ofa normal vector is same as the direction of a registered adjacent normalvector.

The inner product of a vector v15 connecting the barycenter G of apolygon pk as a determination target and the position p15 of the distalend portion 15 when a two-dimensional image used in 3D shape estimationis acquired as illustrated in FIG. 12 and the normal vector vnk of thepolygon pk is calculated, and it is determined that the directions aresame when the value of the inner product is equal to or larger thanzero, or it is determined that the directions are inverted with respectto each other when the value is less than zero. In FIG. 12, an angle θbetween both vectors is smaller than 90°, and the inner product is equalto or larger than zero.

Accordingly, in FIG. 12, for example, the inner surface of a polygonpO4′ at an obtuse angle with respect to the inner surface of an adjacentpolygon (pO3 in FIG. 12) as illustrated with a dotted line cannot beobserved (thus, such a polygon is not generated, and the determinationof the direction of a normal vector is not performed).

In this manner, when the enhanced display is not selected, the 3D modelimage I3 b as illustrated in FIG. 13 is displayed on the monitor 8 in acolor different from a background color.

Most of a luminal organ extending from the ureter on the lower side tothe renal pelvis and calyx on the upper side is drawn with polygons(whereas part of the luminal organ lacks) as illustrated in FIG. 13, andthe (outer) plane of a polygon representing the outer surface of theluminal organ is displayed in a whitish color (for example, green). Notethat the surround of polygons in the 3D model image I3 c is displayed ina background color such as blue.

In FIG. 13, part of the inner surface colored in gray is displayed at alower renal calyx part, and the inner surface colored in gray isdisplayed at a middle renal calyx part above the lower renal calyx part.A boundary is exposed in an upper renal calyx in FIG. 13.

The operator can easily visually recognize, from the 3D model image I3 cin which the inner surface is displayed in a predetermined color in thismanner with a boundary region at the inner surface colored in thepredetermined color, that an unstructured region is not structured norcolored because the region is yet to be observed exists.

In this manner, the 3D model image I3 c displayed as illustrated in FIG.13 is a three-dimensional model image displayed in such a manner thatthe operator can easily visually recognize any unstructured region.

Note that, when the 3D model image I3 c as illustrated in FIG. 13 isgenerated, a partial region of the inner surface which cannot beobserved typically from outside of a closed luminal organ is displayedin a color that allows easy visual recognition, and thus it can bevisually recognized that a region adjacent to the region is anunstructured region that is not observed.

However, like, for example, the upper renal calyx in FIG. 13, when anobserved inner surface is hidden behind an outer surface in front of theinner surface and is not displayed, and has a boundary shape difficultto visually recognize opening of the boundary, it is potentiallyoverlooked that an unstructured region exists in the part. The operatorunderstands the shape of the luminal organ on which observation orexamination is performed, and thus the probability of overlooking islow, but it is desired to reduce a load on the operator as much aspossible to allow the operator to easily and smoothly perform endoscopeexamination.

In the present embodiment, the enhanced display can be selected toachieve the reduction, and processing at steps S23, S24, and S25 in FIG.6 is performed when the enhanced display is selected.

When the enhanced display is selected, the boundary enhancementprocessing section 42 f performs processing of searching for (orextracting) a side of a polygon in a boundary region by usinginformation of a polygon list at step S23.

When the luminal organ as an examination target is the renal pelvis andcalyx 51, the renal pelvis 51 a bifurcates into a plurality of the renalcalyces 51 b. In the example illustrated in FIG. 7, three sides of eachpolygon pi are each shared with an adjacent polygon.

However, a polygon at an edge of a structured region and in a boundaryregion with an unstructured region has a side not shared with any otherpolygon. FIG. 14 schematically illustrates polygons near a boundary, andFIG. 15 illustrates a polygon list corresponding to the polygonsillustrated in FIG. 14.

In FIG. 14, a side e14 of a polygon p12 and a side e18 of a polygon p14indicate a boundary side, and the right side of the sides is anunstructured region. In FIG. 14, the boundary side is illustrated with abold line. In reality, the boundary side typically includes a largernumber of sides. Note that, in FIG. 14, sides e11, e17, and e21 areshared between polygons p11, p13, and p15 and polygons p17, p18, and p19illustrated with dotted lines. Sides e12 and e20 are shared between thepolygons p11 and p15 and polygons p10 and p16 illustrated withdouble-dotted and dashed lines.

In the example illustrated in FIG. 14, the polygon list as illustratedin FIG. 15 is obtained, and in the polygon list, the side e14 of thepolygon p12 and the side e18 of the polygon p14 appear once, and theother sides appear twice. Thus, the polygon processing section 42 cextracts, as a boundary side, any side appearing once from the polygonlist in the processing of searching for (a polygon of) a boundaryregion. In other words, the polygon processing section 42 c extracts, asa boundary side, a side not shared between a plurality of polygons(three-dimensionally adjacent to each other) (that is, a side belongingto only one polygon) in a polygon list as a list of all polygonsrepresenting an observed structured region.

Note that a color used in coloring in accordance with a determinationresult of whether an observed plane of a polygon is an inner surface oran outer surface is set in the rightmost column in the polygon listillustrated in FIG. 15. In FIG. 15, since the inner surface is observed,G representing gray is set.

At the next step S24, the boundary enhancement processing section 42 fproduces a boundary list from the information extracted at the previousstep S23 and notifies the coloring processing section 42 g of theproduction.

FIG. 16 illustrates the boundary list generated at step S24. Theboundary list illustrated in FIG. 16 is a list of a boundary side of anypolygon searched for (extracted) up to the processing at step S23 andappearing only once.

At the next step S25, the coloring processing section 42 g refers to theboundary list and colors any boundary side in a boundary color (forexample, red) that can be easily visually recognized by the user such asan operator. In this case, the thickness of a line drawing a boundaryside may be increased (thickened) to allow easier visual recognition ofthe boundary side in color. In the boundary list illustrated in FIG. 16,the rightmost column indicates an enhancement color (boundary color) inwhich a boundary side is colored by the coloring processing section 42g. In the specific example illustrated in FIG. 16, R representing red iswritten as an enhancement color used in coloring. Alternatively, aboundary region within a distance equal to or smaller than a thresholdfrom a boundary side may be colored in a boundary color or anenhancement color such as red.

Note that the processing of coloring a boundary side is not limited toexecution at step S25, but may be performed in the processing at stepS20 depending on whether the boundary enhancement is selected.

Note that, since the processing illustrated in FIG. 6 repeats similarprocessing in a loop as described above, when the boundary enhancementis selected and a region, an image of which is picked up by the imagepickup section 25 changes with movement of the insertion section 11, apolygon list and a boundary list before the change are updated.

In this manner, when the boundary enhancement is selected, a 3D modelimage I3 d corresponding to FIG. 13 is displayed on the monitor 8 asillustrated in FIG. 17.

In the 3D model image I3 d illustrated in FIG. 17, a boundary side ofeach polygon in a boundary region in the 3D model image I3 c illustratedin FIG. 13 is colored in an enhancement color. As illustrated in FIG.17, a boundary side of each polygon in a structured region, which ispositioned at a boundary with an unstructured region is colored in anenhancement color, and thus the user such as an operator can recognizethe unstructured region adjacent to the boundary side in an easilyvisually recognizable state. Note that FIG. 17 is illustrated inmonochrome display, and thus a boundary side illustrated with a linehaving a thickness larger than the thickness of an outline appears notlargely different from the outline, but the boundary side is displayedin a distinct enhancement color. Thus, when the 3D model image I3 d isdisplayed on the monitor 8 that performs color display, the boundaryside can be visually recognized in a state largely different from thestate of the outline. In the monochrome display, to facilitateidentification of the boundary side from the outline, the boundary sidemay be displayed with a line having a thickness larger than thethickness of the outline by a threshold or more or a line having athickness several times larger than the thickness of a line of theoutline.

In this manner, the endoscope system and the image processing methodaccording to the present embodiment can generate a three-dimensionalmodel image in which an unstructured region is displayed in an easilyvisually recognizable manner.

In the present embodiment, since the 3D model image I3 d in which theboundary between a structured region and an unstructured region isdisplayed in an enhanced manner is generated when the enhanced displayis selected, the user such as an operator can recognize the unstructuredregion in a more easily visually recognizable state.

The following describes a first modification of the first embodiment.The present modification has a configuration substantially same as theconfiguration of the first embodiment, but in processing when theenhanced display is selected, a plane including a boundary side isenhanced instead of the boundary side as in the first embodiment.

FIG. 18 illustrates the contents of processing in the presentmodification. In FIG. 18, the processing of producing (changing) aboundary list at step S24 in FIG. 6 is replaced with processing ofchanging a color of a polygon list, which is described at step S24′, andthe processing of coloring a boundary side at step S25 is replaced withprocessing of coloring a boundary plane at step S25′. Any processingpart different from processing in the first embodiment will be describedbelow.

When the enhanced display is selected at step S19, the processing ofsearching for a boundary is performed at step S23, similarly to thefirst embodiment. In the processing at step S23, a polygon list asillustrated in FIG. 15 is produced, and a polygon having a boundary sideas illustrated in FIG. 16 is extracted.

At the next step S24′, the boundary enhancement processing section 42 fchanges a color in the polygon list including a boundary side to aneasily visually recognizable color (enhancement color) as illustratedin, for example, FIG. 19.

In the polygon list illustrated in FIG. 19, the colors of the polygonsp12 and p14 including the respective boundary sides e14 and e18 in thepolygon list illustrated in FIG. 15 are changed from gray to red.

In simple words, the enhancement color in FIG. 16 is a color forenhancing a boundary side, but in the present modification, theenhancement color is set to a color for enhancing the plane of a polygonincluding the boundary side. Note that, in this case, the plane mayinclude the boundary side in the enhancement color.

At the next step S25′, the boundary enhancement processing section 42 fcolors, in the enhancement color, the plane of the polygon changed tothe enhancement color, and then the process proceeds to the processingat step S20.

FIG. 20 illustrates a 3D model image I3 e generated by the presentmodification and displayed on the monitor 8. In FIG. 20, the color of apolygon (that is, a boundary polygon) including a side adjacent to aboundary is an enhancement color (specifically, red R in FIG. 20). FIG.20 illustrates an example in which a boundary side is also displayed inred in an enhanced manner.

The present modification achieves effects substantially same as theeffects of the first embodiment. More specifically, when the enhanceddisplay is not selected, effects same as the effects of the firstembodiment when the enhanced display is not selected are achieved, andwhen the enhanced display is selected, a boundary plane including aboundary side of a boundary polygon is displayed in an easily visuallyrecognizable enhancement color, and thus the effect of allowing theoperator to easily recognize an unobserved region at a boundary of anobservation region is achieved.

The following describes a second modification of the first embodiment.The present modification has a configuration substantially same as theconfiguration of the first embodiment, but in processing when theenhanced display is selected, processing different from processing inthe first embodiment is performed. In the present modification, theboundary enhancement processing section 42 f in the image generationsection 42 b in FIG. 2 is replaced with an enhancement processingsection (denoted by 42 f) corresponding to selection of the enhanceddisplay (processing result is similar to a result obtained by theboundary enhancement processing section 42 f).

FIG. 21 illustrates processing in the present modification. In FIG. 21,when the enhanced display is not selected, processing same as processingin the first embodiment is performed. When the enhanced display isselected, the enhancement processing section 42 f calculates anycurrently added polygon from a polygon list set after three-dimensionalshape estimation in the last repetition as described at step S41.

Note that the addition is made to a polygon list in a blank state in thefirst processing, and thus the calculation is made on all polygons.

FIG. 22 illustrates the range of additional polygons acquired in thesecond processing for (the range of) hatched polygons acquired in thefirst processing. At the next step S42, the enhancement processingsection 42 f sets an interest region and divides polygons into aplurality of sub blocks.

As illustrated in FIG. 22, the enhancement processing section 42 f sets,for example, a circular interest region centered at an apex (or thebarycenter) of a polygon in the range of additional polygons and dividesthe interest region into, for example, four equally divided sub blocksas illustrated with dotted lines. In reality, for example, a sphericalinterest region is set to a three-dimensional polygon plane, anddivision into a plurality of sub blocks is performed.

FIG. 22 illustrates a situation in which interest regions R1 and R2 areset to respective apexes vr1 and vr2 of interest, the interest region R1is divided into four sub blocks R1 a, R1 b, R1 c, and R1 d, and theinterest region R2 is divided into four sub blocks R2 a, R2 b, R2 c, andR2 d.

At the next step S43, the enhancement processing section 42 f calculatesthe density or number of apexes (or the barycenters) of polygons in eachsub block. The enhancement processing section 42 f also calculateswhether the density or number of apexes (or the barycenters) of polygonshas imbalance between sub blocks.

In the interest region R1, each sub block includes a plurality of apexesof continuously formed polygons and the like, and the density or numberof apexes has small imbalance between sub blocks, whereas in theinterest region R2, the density or number of apexes has large imbalancebetween the sub blocks R2 b and R2 c and between the sub blocks R2 a andR2 d. The sub blocks R2 b and R2 c have values substantially same as thevalue of the sub block R1 a or the like in the interest region R1, butthe sub blocks R2 a and R2 d do not include apexes (or the barycenters)of polygons except at the boundary, and thus have values smaller thanthe values of the sub blocks R2 b and R2 c. The number of apexes haslarge imbalance between the sub blocks R2 b and R2 c and between the subblocks R2 a and R2 d.

At the next step S43, the enhancement processing section 42 f performsprocessing of coloring a polygon satisfying a condition that the densityor number of apexes (or the barycenters) of polygons has imbalance(equal to or larger than an imbalance threshold) between sub blocks andthe density or number of apexes (or the barycenters) of polygons isequal to or smaller than a threshold, or apexes of the polygon in aneasily visually recognizable color (enhancement color such as red). InFIG. 22, for example, the apexes vr2, vr3, and vr4 or polygons sharingthe apexes are colored. After the processing at step S44 or after stepS45 is performed, the process proceeds to the processing at step S20.

When coloring is performed in this manner, the user can perform, throughthe enhanced display selection section 44 b of the input apparatus 44,selection for increasing a coloring range to obtain visibility forachieving easier visual recognition. When the selection for increasingthe coloring range is performed, processing of increasing the coloringrange is performed as described below.

For the processing S44 of coloring a polygon satisfying theabove-described condition (referred to as a first condition) thatimbalance exists in density or the like or any apex of the polygon, theenhancement processing section 42 f further enlarges the coloring rangeat step S45 illustrated with a dotted line in FIG. 21. As describedabove, the processing at step S45 illustrated with a dotted line isperformed when selected.

The enhancement processing section 42 f colors (any apex of) a polygonsatisfying the first condition as described at step S44, but also colors(any apex of) a polygon positioned within a constant distance from (anyapex of) a polygon matching the first condition at step S45 and addedsimultaneously with (the apex of) the polygon matching the firstcondition.

In such a case, for example, the first uppermost polygons in thehorizontal direction or the first and second uppermost polygons in thehorizontal direction in FIG. 22 are colored. The range of polygons to becolored can be increased by increasing the constant distance.

Note that it can be regarded that newly added points (vr2, vr3, and vr4in FIG. 22) around which a boundary exists satisfy a second conditionfor coloring in an easily visually recognizable color.

FIG. 23 illustrates exemplary display of a 3D model image I3 f accordingto the present modification. The 3D model image I3 f is substantiallysame as the 3D model image I3 e illustrated in FIG. 20. Note that FIG.23 omits notation of coloring of a polygon or the like adjacent to aboundary in FIG. 20 in R as an enhancement color. The presentmodification achieves effects substantially same as the effects of thefirst embodiment. That is, when the enhanced display is not selected,effects same as the effects of the first embodiment when the enhanceddisplay is not selected are achieved, and when the enhanced display isselected, a boundary region of any structured polygon can be displayeddistinct in an easily visually recognizable color, similarly to thefirst embodiment when the enhanced display is selected. Thus, anyunstructured region positioned adjacent to the boundary region and yetto be observed can be easily recognized.

The following describes a third modification of the first embodiment.

The present modification corresponds to a case in which display similarto display when the enhanced display is selected is performed even whenthe enhanced display is not selected in the first embodiment.

Accordingly, the present modification corresponds to a configuration inwhich the input apparatus 44 does not include the enhanced displayselection section 44 b in the configuration illustrated in FIG. 2, andthe boundary enhancement processing section 42 f does not need to beprovided, but processing similar to processing performed by the boundaryenhancement processing section 42 f is performed effectively. The otherconfiguration is substantially same as the configuration of the firstembodiment.

FIG. 24 illustrates the contents of processing in the presentmodification. The flowchart in FIG. 24 illustrates processing similar tothe flowchart illustrated in FIG. 6, and thus the following descriptionwill be made only on any different part of the processing.

Steps S1 to S18 are processing same as the corresponding processing inFIG. 6, and after the processing at step S18, the polygon processingsection 42 c performs the processing of searching for an unobservedregion at step S51.

As described above, the three-dimensional shape estimation is performedat step S13 and the processing of generating a 3D model image isperformed through processing of bonding polygons to the surface of anobserved region, but when an unobserved region exists as an openingportion in, for example, a circle shape (adjacent to the observedregion) at a boundary of the observed region, processing performed on aplane in the observed region is potentially performed on the openingportion by bonding polygons to the opening portion.

Thus, in the present modification, in the processing of searching for anunobserved region at step S51, an angle between the normal of a polygonset to a region of interest and the normal of a polygon positionedadjacent to the polygon and set in the observed region is calculated,and whether the angle is equal to or larger than a threshold ofapproximately 90° is determined.

At the next step S52, the polygon processing section 42 c extractspolygons, the angle between the two normals of which is equal to orlarger than the threshold.

FIG. 25 illustrates an explanatory diagram of an operation in thepresent modification. FIG. 25 illustrates an exemplary situation inwhich polygons are set at an observed luminal shape part extending inthe horizontal direction, and a substantially circular opening portion Oas an unobserved region exists at the right end of the part.

In this case, similarly to a case in which polygons are set in theobserved region adjacent to the opening portion O, processing of settingpolygons to the opening portion O is potentially performed. In such acase, the angle between a normal Ln1 of a polygon set in the observedregion adjacent to a boundary of the opening portion O and a normal Lo1of a polygon pO1 positioned adjacent to the polygon and set to block theopening portion O is significantly larger than the angle between twonormals Lni and Lni+1 set to two polygons adjacent to each other in theobserved region, and is equal to or larger than a threshold.

FIG. 25 illustrates, in addition to the normals Ln1 and Lo1, a normalLn2 and a normal Lo2 of a polygon pO2 set to block the opening portionO.

At the next step S53, the coloring processing section 42 g colors, in acolor (for example, red) different from a color for the observed region,a plurality of polygons (polygons pO1 and pO2 in FIG. 25), the anglebetween the two normals of which is equal to or larger than thethreshold, and a polygon (polygon pO3 between the polygons pO1 and pO2)surrounded by a plurality of polygons. After the processing at step S53,the process proceeds to the processing at step S20.

FIG. 26 illustrates a 3D model image I3 g according to the presentmodification. In FIG. 26, an unobserved region is displayed in red.

According to the present modification, when a polygon is set adjacent toa polygon in an observed region and set in an unobserved region, thepolygon can be colored to facilitate visual recognition of theunobserved region.

The following describes a fourth modification of the first embodiment.

The present modification allows easy recognition of an unobserved regionby simplifying the shape of the boundary between an observed region andthe unobserved region (to reduce the risk of false recognition that, forexample, a complicated shape is attributable to noise).

In the present modification, in the configuration illustrated in FIG. 2,the input apparatus 44 includes a smoothing selection section (denotedby 44 c) for selecting smoothing in place of the enhanced displayselection section 44 b, and the image generation section 42 b includes asmoothing processing section (denoted by 42 h) configured to performsmoothing processing in place of the boundary enhancement processingsection 42 f. The other configuration is substantially same as theconfiguration of the first embodiment.

FIG. 27 illustrates the contents of processing in the presentmodification. The processing illustrated in FIG. 27 is similar to theprocessing illustrated in FIG. 6, and thus the following descriptionwill be made only on any different part of the processing.

In the processing illustrated in FIG. 27, the processing at step S19 inFIG. 6 is replaced with processing of determining whether to selectsmoothing at step S61.

Smoothing processing at step S62 is performed after the boundary searchprocessing at step S23, and boundary search processing is furtherperformed at step S63 after the smoothing processing, thereby producing(updating) a boundary list.

In the present modification, to display the shape of the boundarybetween an observed region and an unobserved region in a simplifiedmanner as described above, a polygon list before the smoothingprocessing at step S62 is performed is held in, for example, theinformation storage section 43, and a held copy is set to a polygon listand used to generate a 3D model image (the copied polygon list ischanged by smoothing, but the polygon list before the change is held inthe information storage section 43).

In the processing at step S61 in FIG. 27, when smoothing is notselected, the process proceeds to step S20 where the processingdescribed in the first embodiment is performed.

When smoothing is selected, the polygon processing section 42 c performsthe processing of searching for a boundary at step S23.

The processing of searching for a boundary at step S23 is described withreference to, for example, FIGS. 14 to 16. Through the processing ofsearching for a boundary, a polygon boundary is extracted as illustratedin, for example, FIG. 28 in some cases. FIG. 28 schematicallyillustrates a situation in which a polygon boundary part of the luminalshape illustrated in FIG. 25 has a complicated shape including unevenportions.

At the next step S62, the smoothing processing section 42 h performssmoothing processing. The smoothing processing section 42 h applies, forexample, a least-square method to calculate a curved surface Pl (theamount of change in the curvature of which is restricted in anappropriate range), the distances of which from the barycenters (orapexes) of a plurality of polygons in a boundary region are minimized.When the degree of unevenness between adjacent polygons is large, thepresent invention is not limited to application of the least-squaremethod to all polygons adjacent to the boundary, but the least-squaremethod may be applied only to some of the polygons.

In addition, the smoothing processing section 42 h performs processingof deleting any polygon part outside of the curved surface Pl. In FIG.28, deleted polygon parts are hatched.

At the next step S63, the smoothing processing section 42 h (or thepolygon processing section 42 c) searches for a polygon forming aboundary region in processing corresponding to the above-describedprocessing (steps S23, S62, and S63). For example, processing ofsearching for a polygon (for example, a polygon pk denoted by areference sign) partially deleted by the curved surface Pl, and for apolygon pa, a side of which is adjacent to the boundary as illustratedin FIG. 28 is performed.

Then, at the next step S64, a boundary list in which sides of thepolygons extracted through the search processing are set as boundarysides is produced (updated). In this case, an apex is newly added to apolygon partially deleted by the curved surface Pl so that the shape ofthe polygon becomes a triangle, and then the polygon is divided. Notethat boundary sides of the polygon pk in FIG. 28 are sides ek1 and ek2partially deleted by the curved surface Pl and a side ep as the curvedsurface Pl. In this case, the side ep as the curved surface Pl isapproximated with a straight side connecting both ends in the plane ofthe polygon pk.

At the next step S25, the coloring processing section 42 g performsprocessing of coloring, in an easily visually recognizable color, theboundary sides of polygons written in the boundary list, and thereafter,the process proceeds to the processing at step S20.

FIG. 29 illustrates a 3D model image I3 h generated in this manner anddisplayed on the monitor 8. According to the present modification, anyboundary part having a complicated shape is displayed as a simplifiedboundary side in an easily visually recognizable color, therebyfacilitating recognition of an unobserved region.

Note that processing may be performed by the following method instead ofthe polygon division by the curved surface Pl.

At step S62, the smoothing processing section 42 h searches for an apexoutside of the curved surface Pl. At the next step S63, the smoothingprocessing section 42 h (or the polygon processing section 42 c)performs processing of deleting a polygon including an apex outside ofthe curved surface Pl from a copied polygon list. At the next step S63,in processing corresponding to the above-described processing (stepsS23, S62, and S63), the smoothing processing section 42 h (or thepolygon processing section 42 c) performs the processing of deleting apolygon including an apex outside of the curved surface Pl from thecopied polygon list, and performs the boundary search described inanother modification.

The following describes a fifth modification of the first embodiment.

In the first embodiment, when the enhanced display is selected, theprocessing of extracting a side of a polygon in a boundary region as aboundary side and coloring the boundary side in a visually recognizablemanner is performed, but in the present modification, when athree-dimensional shape is expressed with points (corresponding to, forexample, points at the barycenters of polygons or apexes of thepolygons) instead of the polygons, processing of extracting, as boundarypoints, points at a boundary in place of boundary sides (of thepolygons) is performed, and processing of coloring the boundary pointsin an easily visually recognizable manner is performed.

Thus, in the present modification, the boundary enhancement processingsection 42 f performs processing of enhancing a boundary point in theconfiguration illustrated in FIG. 2. FIG. 30A illustrates theconfiguration of an image processing apparatus 7′ in the presentmodification. The image processing apparatus 7′ in the presentmodification does not perform, for example, processing of displaying athree-dimensional shape with polygons, and thus does not include thepolygon processing section 42 c and the inner and outer surfacedetermination section 42 e illustrated in FIG. 2. The otherconfiguration is substantially same as the configuration of the firstembodiment.

FIG. 30B illustrates the contents of processing in the presentmodification. The flowchart in FIG. 30B illustrates processing similarto the flowchart illustrated in FIG. 6, and thus the followingdescription will be made only on any different part of the processing.In the flowchart illustrated in FIG. 30B, the processing at steps S15 toS20 in FIG. 6 is not performed. Thus, the process proceeds to theprocessing at steps S23 and S24 after the processing at step S14,performs processing of coloring a boundary point as described at stepS71 in place of the processing of coloring a boundary side at step S25in FIG. 6, and proceeds to the processing at step S21 after theprocessing at step S71. However, as described below, for example, thecontents of the processing of producing (changing) a boundary list atstep S24 same as step S24 in FIG. 6 is slightly different from thecontents of processing in the first embodiment.

At step S23, in processing of searching for a boundary and extracting aboundary point, the boundary enhancement processing section 42 f mayextract a boundary point through the processing (processing ofsatisfying at least one of the first condition and the second condition)described with reference to FIG. 22 in the second modification.

That is, as for the first condition, a plurality of interest regions areset to a point (barycenter or apex) of interest, the density of pointsor the like in a sub block of each interest region is calculated, andany point satisfying a condition that the density or the like hasimbalance and the density has a value equal to or smaller than athreshold is extracted as a boundary point.

Alternatively, as for the second condition, a newly added point aroundwhich a boundary exists is extracted as a boundary point. In the exampleillustrated in FIG. 22, vr2, vr3, vr4, and the like are extracted asboundary points.

FIG. 31 illustrates a 3D model image I3 i generated by the presentmodification and displayed on the monitor 8. As illustrated in FIG. 31,points in boundary regions are displayed in an easily visuallyrecognizable color. Note that a point in a boundary region may becolored as a bold point (having an increased area) in an easily visuallyrecognizable color (enhancement color). In addition, a middle pointbetween two adjacent points in a boundary region may be displayed in aneasily visually recognizable color.

According to the present modification, a point at the boundary betweenan observed structured region and an unobserved unstructured region isdisplayed in an easily visually recognizable color, and thus theunstructured region can be easily recognized. Note that a line (referredto as a border line) connecting the above-described adjacent boundarypoints may be drawn and colored in an easily visually recognizable colorby the coloring processing section 42 g. In addition, any point includedwithin a distance equal to or smaller than a threshold from a boundarypoint may be colored as a bold point (having an increased area) in aneasily visually recognizable color (enhancement color).

Note that a three-dimensional shape can be displayed with thebarycenters of observed polygons in the present modification. In thiscase, processing of calculating the barycenters of polygons isperformed. The processing may be applied to a sixth modificationdescribes below.

In the processing at step S71 in FIG. 30B according to the fifthmodification, any surrounding point near a boundary point may beadditionally colored in an easily visually recognizable color togetherwith the boundary point (refer to FIG. 33). The sixth modification ofthe first embodiment in which a processing result substantially same asa processing result in this case will be described next.

In the sixth modification, a boundary point and any surrounding pointaround the boundary point in the fifth modification are colored andenhanced in an easily visually recognizable color, and a configurationsame as the configuration according to the fifth modification isemployed.

FIG. 32 illustrates the contents of processing in the presentmodification. The processing illustrated in FIG. 32 is similar to theprocessing according to the fifth modification of the first embodimentillustrated in FIG. 30B, and processing at steps S81 to S83 is performedafter the processing at step S14, and the process proceeds to theprocessing at step S21 after the processing at step S83. After theprocessing at step S14, the boundary enhancement processing section 42 fperforms processing of calculating any added point since the lastrepetition as described at step S81.

The example of range of added points is same as, for example, the rangein the case with polygons described with reference to FIG. 22. At thenext step S82, the boundary enhancement processing section 42 f changesthe color of any newly added point in a point list that is a list ofadded points to a color (for example, red) different from an observedcolor. The boundary enhancement processing section 42 f also performsprocessing of setting, back to the observed color, the color of a pointin the different color at a distance equal to or larger than a thresholdfrom a newly added point in the point list.

At the next step S83, the coloring processing section 42 g performsprocessing of coloring points of polygons in accordance with colorswritten in the polygon list up to the previous step S82, and then theprocess proceeds to the processing at step S21.

FIG. 33 illustrates a 3D model image I3 j according to the presentmodification. In addition to boundary points in the example illustratedin FIG. 31, points around the boundary points are colored and displayedin the same color, which makes it easier for the operator to check anunobserved region.

For example, only an unobserved region may be displayed in accordancewith an operation of the input apparatus 44 by the user. When anyobserved region is not displayed, the operator can easily check anunobserved region behind the observed region. Note that the function ofdisplaying only an unobserved region may be provided to any otherembodiment or modification.

The following describes a seventh modification of the first embodiment.In the present modification, an index indicating an unobserved region isadded and displayed, for example, when index addition is selected in thefirst embodiment. FIG. 34 illustrates the configuration of an imageprocessing apparatus 7B in the present modification.

In the image processing apparatus 7B, the input apparatus 44 in theimage processing apparatus 7 illustrated in FIG. 2 includes an indexdisplay selection section 44 d for selecting index display, and theimage generation section 42 b in the image processing apparatus 7illustrated in FIG. 2 includes an index addition section 42 i configuredto add an index to an unobserved region. The other configuration is sameas the configuration of the first embodiment. FIG. 35 illustrates thecontents of processing in the present modification.

The flowchart illustrated in FIG. 35 has contents of processing inwhich, in addition to the flowchart illustrated in FIG. 6, processingfor displaying an index in accordance with a result of the index displayselection is additionally performed.

When the enhanced display is selected at step S19, after the processingat steps S23 and S24 is performed, the control section 41 determineswhether index display is selected at step S85. When index display is notselected, the process proceeds the processing at step S25, or when indexdisplay is selected, the index addition section 42 i performs processingof calculating an index to be added and displayed at step S86, and then,the process proceeds to the processing at step S25.

The index addition section 42 i

a. calculates a plane including a side at a boundary,b. subsequently calculates the barycenter of a point at the boundary,andc. subsequently calculates a point on a line parallel to the normal ofthe plane calculated at “a” and at a constant distance from thebarycenter of the point at the boundary and adds an index.

FIG. 36 illustrates a 3D model image I3 k in this case. FIG. 36 is adiagram in which indexes are added to the 3D model image I3 dillustrated in FIG. 17.

When the enhanced display is not selected at step S19 in FIG. 35, thecontrol section 41 determines whether index display is selected at stepS87. When index display is not selected, the process proceeds to theprocessing at step S20, or when index display is selected, similarly tostep S23, the processing of searching for a boundary is performed atstep S88, and then, the index addition section 42 i performs processingof calculating an index to be added and displayed at step S89, beforethe process proceeds to the processing at step S20.

FIG. 37 illustrates a 3D model image I31 in this case. FIG. 37 is adiagram in which indexes are added to the 3D model image I3 cillustrated in FIG. 13. Note that the indexes are colored in, forexample, yellow.

According to the present modification, selection for displaying the 3Dmodel images I3 c and I3 d as in the first embodiment can be performed,and also, selection for displaying the 3D model images I31 and I3 k towhich indexes are added can be performed. Indexes may be displayed on 3Dmodel images I13 e, I13 f, I13 g, I13 h, I13 i, and I13 j byadditionally performing the same processing.

The following describes an eighth modification of the first embodiment.The seventh modification describes the example in which an indexillustrating a boundary or an unobserved region with an arrow isdisplayed outside of the 3D model images I3 c and I3 d. Alternatively,index display in which light from a light source set inside a lumen in a3D model image leaks out of an opening portion as an unobserved regionmay be performed as described below.

In processing according to the present modification, only the processingof calculating an index at step S86 or S89 in FIG. 36 in the seventhmodification is replaced with processing of generating an indexillustrated in FIG. 39. Note that the index addition section 42 ifunctions as an opening portion extraction section configured to extractan opening portion as an unstructured region having an area equal to orlarger than predetermined area when processing described below withreference to FIG. 38 or the like is performed, and a light sourcesetting section configured to set a point light source at a position ona normal extending on the internal side of a lumen.

FIG. 38 illustrates the contents of processing of generating an index inthe present modification.

When the processing of generating an index is started, the indexaddition section 42 i calculates an opening portion as an unobservedregion that has an area equal to or larger than a defined area at thefirst step S91. FIG. 39 illustrates an explanatory diagram for theprocessing illustrated in FIG. 38, and illustrates an opening portion 61as an unobserved region that has an area equal to or larger than adefined area (or predetermined area) in a luminal organ.

At the next step S92, the index addition section 42 i sets (on theinternal side of the lumen) a normal 62 from the barycenter of pointsincluded in the opening portion 61. As illustrated in a diagram on theright side in FIG. 39, the normal 62 is a normal to a plane passingthrough a total of three points of a barycenter 66, a point 67 nearestto the barycenter 66, and a point 68 farthest from the barycenter 66among the points included in the opening portion 61, and has a unitlength from the barycenter 66. The direction of the normal is adirection in which a large number of polygons forming a 3D model exist.Note that three representative points set on the opening portion 61 asappropriate may be employed in place of the above-described threepoints.

At the next step S93, the index addition section 42 i sets a point lightsource 63 at a defined length (inside the lumen) along the normal 62from the barycenter 66 of the opening portion 61.

At the next step S94, the index addition section 42 i draws linesegments 64 extending from the point light source 63 toward the outsideof the opening portion 61 through (respective points on) the openingportion 61.

At the next step S95, the index addition section 42 i colors the linesegments 64 in the color (for example, yellow) of the point light source63. Display with added indexes may be performed by performing processingas described below in addition to the processing illustrated in FIG. 38.Steps S91 to S93 illustrated in FIG. 38 are same in the processingdescribed below.

At a step following step S93, as illustrated in an uppermost diagram inFIG. 40, line segments (line segments illustrated with a dotted lines)64 a connecting the point light source 63 and two points facing to eachother and sandwiching the barycenter 66 of the opening portion 61 aredrawn, and a region (hatched region) of a polygon connecting linesegments (line segments illustrated with solid lines) 65 b extendingfrom the two points toward the outside of the opening portion 61 and aline segment connecting the two points is colored in the color of thepoint light source and set as an index 65. In other words, the index 65is formed by coloring, in the color of the point light source 63, aregion outside the opening portion 61 within the angle between two linesegments passing through the two points on the opening portion 61 facingto each other and sandwiching the barycenter 66 from the point lightsource 63.

Note that, when a Z axis is defined to be an axis orthogonal to adisplay screen and an angle θ between the normal 62 and the Z axis isequal to or smaller than a certain angle (for example, 45 degrees) asillustrated in a lowermost diagram in FIG. 40, the inside of the openingportion 61, which is hatched with bold lines, is colored and displayed.

FIG. 41 illustrates a 3D model image I3 m when the enhanced display andthe index display are selected in the present modification.

As illustrated in FIG. 41, in addition to the enhanced display, theindex (part hatched in FIG. 41) 65 as if light leaks from an openingadjacent to an unobserved region is displayed to indicate the unobservedregion, and thus a situation in which an unobserved region equal to orlarger than the defined area exists can be recognized in an easilyvisually recognizable state.

The following describes a ninth modification of the first embodiment. Inthe first embodiment and the modifications described above, a 3D modelimage viewed in a predetermined direction is generated and displayed asillustrated in, for example, FIGS. 13, 17, 20, and 23.

FIG. 42 illustrates the configuration of an image processing apparatus7C in the present modification.

In the present modification, the image generation section 42 b furtherincludes a rotation processing section 42 j configured to rotate a 3Dmodel image, and a region counting section 42 k configured to count thenumber of boundaries (regions), unobserved regions, or unstructuredregions in addition to the configuration illustrated in FIG. 2 in thefirst embodiment.

The rotation processing section 42 j rotates a 3D model image viewed ina predetermined direction around, for example, a core line so that, whenthe 3D model image viewed in a predetermined direction is a front image,the front image and a back image viewed from a back surface on a sideopposite to the predetermined direction can be displayed side by side,and 3D model images viewed in a plurality of directions selected by theoperator can be displayed side by side. In addition, overlooking of aboundary can be prevented.

For example, when the number of unstructured regions counted by theregion counting section 42 k is zero in a front image viewed in apredetermined direction, a 3D model image may be rotated by the rotationprocessing section 42 j so that the number is equal to or larger thanone (except for a case in which unstructured regions exist nowhere).When an unstructured region in three-dimensional model data cannot bevisually recognized, the image generation section 42 b may provide thethree-dimensional model data with rotation processing, generate athree-dimensional model image in which the unstructured region isvisually recognizable, and display the three-dimensional model image.

In place of, for example, the 3D model image I3 d in which a boundary(or unobserved region) appearing on a front side when viewed in apredetermined direction is displayed in an enhanced manner, a back-sideboundary Bb appearing when viewed from a back side may be illustratedwith a dotted line in a color (for example, purple; note that abackground color is light blue and thus distinguishable from purple)different from a color (for example, red) indicating a boundaryappearing on the front side in a 3D model image I3 n in the presentmodification as illustrated in FIG. 43A.

In the 3D model image I3 o, a count value of discretely existingboundaries (regions) counted by the region counting section 42 k may bedisplayed in the display screen of the monitor 8 (in FIG. 43A, the countvalue is four).

In the display illustrated in FIG. 43A, a boundary appearing on the backside, which does not appear when viewed in a predetermined direction(front), is displayed in a color different from a color illustrating aboundary on the front side to prevent overlooking of any boundary on theback side, and also the count value is displayed to effectively preventoverlooking of a boundary. In addition, effects same as the effects ofthe first embodiment are achieved.

Note that only a boundary or a boundary region may be displayed withoutdisplaying an observed 3D model shape. For example, only four boundaries(regions) in FIG. 43A may be displayed. In such a case, the boundaries(regions) are displayed floating in a space in an image. Alternatively,the outline of a 3D model shape may be displayed with, for example, adouble-dotted and dashed line, and boundaries (regions) may be displayedon the outline of the 3D model shape, thereby displaying the positionsand boundary shapes of the boundaries (regions) in the 3D shape in aneasily recognizable manner Such display can effectively prevent boundaryoverlooking.

A 3D model image may be rotated and displayed as described below.

When it is sensed that an unstructured region is disposed andsuperimposed behind (on the back surface of) a structured region on thesurface of the monitor 8 when viewed by the user and thus cannot bevisually recognized by the user, the rotation processing section 42 jmay automatically rotate the 3D model image so that the unstructuredregion is disposed on the front side at which the unstructured region iseasily visually recognizable.

When a plurality of unstructured regions exist, the rotation processingsection 42 j may automatically rotate the 3D model image so that anunstructured region having a large area is disposed on the front side.

For example, a 3D model image I3 n-1 as a rotation processing targetillustrated in FIG. 43B may be rotated and displayed so that anunstructured region having a large area is disposed on the front side asillustrated in FIG. 43C. Note that FIGS. 43B and 43C illustrate a statein which an endoscope image and the 3D model image I3 n−1 are disposedon the right and left sides on the display screen of the monitor 8. The3D shape of a renal pelvis and calyx modeled and displayed in the 3Dmodel image I3 n−1 is illustrated on the right side of the displayscreen.

When a plurality of unstructured regions exist, the rotation processingsection 42 j may automatically rotate a 3D model image so that anunstructured region nearest to the leading end position of the endoscope2I is disposed on the front side.

Note that the unstructured region may be displayed in an enlargedmanner. An unobserved region may be largely displayed in an enlargedmanner to display the unstructured region in an easily visuallyrecognizable manner.

For example, when an unstructured region Bu1 exists behind (on the backside) as illustrated with a dotted line in FIG. 43D, the unstructuredregion may be (partially) displayed in a visually recognizable manner bydisplaying an unstructured region Bu2 having an enlarged size largerthan the size of a structured region part on the front side covering theunstructured region Bu1.

Note that not only an unstructured region behind (on the back side) butall unstructured regions may be displayed in an enlarged manner todisplay the unstructured region in a more easily visually recognizablemanner.

The following describes a tenth modification of the first embodiment.FIG. 44 illustrates an image processing apparatus 7D in the tenthmodification. In the present modification, the image generation section42 b in the image processing apparatus 7C in the ninth modificationillustrated in FIG. 42 further includes a size calculation section 42 lconfigured to calculate the size of an unstructured region. The sizecalculation section 42 l functions as a determination section 42 mconfigured to determine whether the size of the unstructured region isequal to or smaller than a threshold. Note that the determinationsection 42 m may be provided outside of the size calculation section 42l. The other configuration is same as the configuration in the ninthmodification.

The size calculation section 42 l in the present modification calculatesthe size of the area of each unstructured region counted by the regioncounting section 42 k. Then, when the calculated size of theunstructured region is equal to or smaller than the threshold,processing of displaying (a boundary of) the unstructured region in anenhanced manner so that (the boundary of) the unstructured region iseasily visually recognizable is not performed, and the unstructuredregion is not counted in the number of unstructured regions.

FIG. 45 illustrates 3D shape data including a boundary B1 having a sizeequal to or smaller than the threshold and a boundary B2 having a sizeexceeding the threshold. The boundary B2 is displayed in an enhancedmanner in an easily visually recognizable color (for example, red) suchas red, whereas the boundary B1 is a small area that does not need to beobserved and thus is not provided with the enhancement processing or isprovided with processing of blocking an opening at the boundary withpolygons (or processing of blocking the opening with polygons to producea pseudo observation region). In other words, an unstructured regionincluding the boundary B1 having a size equal to or smaller than thethreshold is not provided with processing of allowing visual recognitionnor processing of facilitating visual recognition.

In the present modification, when the determination section 42 mdetermines whether to perform the enhancement processing, thedetermination is not limited to a condition based on whether the area ofan unstructured region or a boundary is equal to or smaller than thethreshold as described above, but the determination may be made based onconditions described below.

That is, the determination section 42 m does not perform the enhancementprocessing or generates a pseudo observed region when at least one ofconditions A to C below is satisfied:

A. when the length of a boundary is equal to or smaller than a lengththreshold,B. when the number of apexes included in the boundary is equal to orsmaller than a threshold for the number of apexes, orC. when in primary component analysis of the coordinates of theboundary, the difference between the maximum and minimum of a secondprimary component or the difference between the maximum and minimum of athird primary component is equal to or smaller than a componentthreshold.

FIG. 46 illustrates an explanatory diagram of the condition C. FIG. 46illustrates 3D shape data of a lumen having a boundary B in acomplicated shape at the right end, an axis A1 of a first primarycomponent is aligned with a longitudinal direction of the lumen, an axisA2 of the second primary component is aligned with a directionorthogonal to the axis A1 of the first primary component in the sheet ofFIG. 46, and an axis A3 of the third primary component is aligned with adirection orthogonal to the sheet.

Subsequently, the coordinates of a boundary are projected onto a planeorthogonal to the axis A1 of the first primary component. FIG. 47illustrates a diagram of the projection. The determination section 42 mcalculates lengths in directions parallel to respective axes of a planeillustrated in FIG. 47 and determines whether the difference between themaximum and minimum of the second primary component or the differencebetween the maximum and minimum of the third primary component is equalto or smaller than a component threshold. FIG. 47 illustrates a maximumlength L1 of the second primary component and a maximum length L2 of thethird primary component.

In the present modification, the effects of the ninth modification areachieved, and in addition, unnecessary display is not performed by notdisplaying a small boundary that does not need to be observed.

The following describes an eleventh modification of the firstembodiment. FIG. 48 illustrates an image processing apparatus 7E in theeleventh modification. In addition to the image processing apparatus 7illustrated in FIG. 2, the image processing apparatus 7E illustrated inFIG. 48 further includes a core line generation section 42 n configuredto generate a core line for 3D shape data. The input apparatus 44includes a core line display selection section 44 e configured todisplay a 3D model image with a core line.

In the present modification, processing same as processing in the firstembodiment is performed when the input apparatus 44 does not performselection for displaying a 3D model image with a core line by the coreline display selection section 44 e, or processing illustrated in FIG.49 is performed when selection for display with a core line by the coreline display selection section 44 e is performed.

The following describes the processing illustrated in FIG. 49. When theprocessing illustrated in FIG. 49 is started, the image processingsection 42 acquires a 2D image from the video processor 4 at step S101,and structures a 3D shape from 2D images inputted in a temporallysubstantially continuous manner. In a specific method, the 3D shape canbe formed from the 2D images through processing same as steps S11 to S20in FIG. 6 described above (by, for example, the method of marchingcubes).

When it is determined that switching to a core line production mode ismade at step S102, the 3D shape structuring is ended to transition tothe core line production mode. The switching to the core line productionmode is determined based on, for example, inputting through operationmeans by the operator or determination of the degree of progress of the3D shape structuring by a processing apparatus.

After the switching to the core line production mode, a core line of theshape produced at step S101 is produced at step S103. Note that coreline production processing can employ publicly known methods such asmethods described in, for example, “Masahiro YASUE, Kensaku MORI,Toyofumi SAITO, et al., Thinning Algorithms for Three-Dimensional GrayImages and Their Application to Medical Images with ComparativeEvaluation of Performance, Journal of The Institute of Electronics,Information and Communication Engineers, J79-D-H(10):1664-1674, 1996”,and “Toyofumi SAITO, Satoshi BANJO, Jun-ichiro TORIWAKI, An Improvementof Three Dimensional Thinning Method Using a Skeleton Based on theEuclidean Distance Transformation —A Method to Control SpuriousBranches-, Journal of The Institute of Electronics, Information andCommunication Engineers, (J84-D2:1628-1635) 2001”.

After the core line is produced, the position of an intersection pointbetween the core line and a perpendicular line extending toward the coreline from a colored region in a different color illustrating anunobserved region in the 3D shape is derived at step S104. Thederivation is simulated in FIG. 50. In FIG. 50, Rm1 and Rm2 (coloredregions hatched in FIG. 50) illustrating unobserved regions exist on the3D shape. Perpendicular lines extend from the unobserved regions Rm1 andRm2 toward the core line already formed at step S103 and illustratedwith a dotted line. Intersection points between the perpendicular linesand the core line are indicated by line segments L1 and L2 on the coreline illustrated with solid lines. Then, at step S105, the line segmentsL1 and L2 are each colored in a color (for example, red) different fromthe color of the other region of the core line.

Through the processing performed so far, the core line illustrating anobserved region and an unobserved region in a pseudo manner is displayed(step S106).

After the formation and the display of the core line, the core lineproduction mode is ended (step S107).

Subsequently, at step S108, the observation position and sight linedirection estimation processing section estimates an observationposition and a sight line direction of the endoscope based on acquiredobservation position and sight line direction data.

In addition, calculation on movement of the observation position ontothe core line is performed at step S109 to illustrate the observationposition estimated at step S108 on the core line in a pseudo manner. Atstep S109, the estimated observation position is moved to a point on thecore line at which the distance between the estimated observationposition and the core line is minimized.

At step S110, the pseudo observation position estimated at step S109 isdisplayed together with the core line. Accordingly, the operator candetermine whether an unobserved region is approached.

The display is repeated from step S108 until determination to end anexamination is made (step S111).

FIG. 51 illustrates an exemplary state when step S106 is ended, andillustrates a core line image Ic generated in the observation regionincluding the unobserved regions Rm1 and Rm2 In FIG. 51, a core line 71and a core line including a line segment 72 are displayed in colorsdifferent from each other, and the user such as an operator can easilyvisually recognize that an unobserved region exists based on the coreline including the line segment 72.

An image processing apparatus having the functions in the firstembodiment to the eleventh modification described above may be provided.FIG. 52 illustrates an image processing apparatus 7G in a twelfthmodification having such functions. The respective components of theimage generation section 42 b and the input apparatus 44 in the imageprocessing apparatus 7G illustrated in FIG. 52 are already described,and thus any duplicate description will be omitted. According to thepresent modification, the user such as an operator has an increasednumber of options for selecting the display format of a 3D model imagewhen displayed on the monitor 8, and in addition to the above-describedeffects, a 3D model image that satisfies a wider range of requirement bythe user can be displayed.

Note that, in the first embodiment including the above-describedmodifications, the endoscope 2A and the like are not limited to aflexible endoscope including the flexible insertion section 11 but arealso applicable to a rigid endoscope including a rigid insertionsection.

The present invention is applicable to, in addition to a case of amedical endoscope used in the medical field, a case in which the insideof, for example, a plant is observed and examined by using an industrialendoscope used in the industrial field.

Parts of the embodiment including the above-described modifications maybe combined to achieve a different embodiment. In addition, only theenhanced display may be performed without coloring the inner surface(inner wall surface or inner wall region) and the outer surface (outerwall surface or outer wall region) of a polygon in different colors.

A plurality of claims may be integrated into one claim, and the contentsof one claim may be divided into a plurality of claims.

What is claimed is:
 1. An image processing apparatus comprising: athree-dimensional model structuring section configured to generate, whenan image pickup signal related to a region in a subject is inputted froman image pickup apparatus configured to pick up an image of an inside ofthe subject, three-dimensional data representing a shape of the regionbased on the image pickup signal; and an image generation sectionconfigured to perform, on the three-dimensional data generated by thethree-dimensional model structuring section, processing of allowingvisual recognition of a boundary region between a structured region thatis a region, an image of which is picked up by the image pickupapparatus, and an unstructured region that is a region, an image ofwhich is yet to be picked up by the image pickup apparatus, and generatea three-dimensional image.
 2. The image processing apparatus accordingto claim 1, wherein when a first image pickup signal related to a firstregion in the subject is inputted from the image pickup apparatus, thethree-dimensional model structuring section generates three-dimensionaldata representing a shape of the first region based on the first imagepickup signal, the image generation section generates athree-dimensional image based on the three-dimensional data representingthe shape of the first region and outputs the three-dimensional image toa display section, when a second image pickup signal related to a secondregion including a region different from the first region is inputtedfrom the image pickup apparatus after the first image pickup signal isinputted, the three-dimensional model structuring section generatesthree-dimensional data representing a shape of the second region basedon the second image pickup signal, and the image generation sectiongenerates three-dimensional images of the first region and the secondregion based on the three-dimensional data representing the shape of thesecond region and outputs the three-dimensional images to the displaysection.
 3. The image processing apparatus according to claim 2, whereinthe three-dimensional model structuring section sets the second imagepickup signal to be, among image pickup signals inputted from the imagepickup apparatus after the first image pickup signal, an image pickupsignal in which a predetermined change amount is detected for the firstregion included in the first image pickup signal.
 4. The imageprocessing apparatus according to claim 2, wherein the image generationsection generates a three-dimensional image by synthesizing thethree-dimensional data representing the shape of the first region andthe three-dimensional data representing the shape of the second regionand outputs the three-dimensional image to the display section.
 5. Theimage processing apparatus according to claim 2, wherein thethree-dimensional model structuring section stores three-dimensionaldata generated based on the first image pickup signal and representingthe shape of the first region in a storage section and additionallystores three-dimensional data generated based on the second image pickupsignal and representing the shape of the second region in the storagesection, and the image generation section generates a three-dimensionalimage by synthesizing the three-dimensional data representing the shapeof the first region and the three-dimensional data representing theshape of the second region that are stored in the storage section, andoutputs the three-dimensional image to the display section.
 6. The imageprocessing apparatus according to claim 2, wherein the three-dimensionalmodel structuring section stores the first image pickup signal in astorage section instead of generating the three-dimensional datarepresenting the shape of the first region when the first image pickupsignal is inputted, and stores the second image pickup signal in thestorage section instead of generating the three-dimensional datarepresenting the shape of the second region when the second image pickupsignal is inputted, and the image generation section generates athree-dimensional image based on the first image pickup signal and thesecond image pickup signal stored in the storage section and outputs thethree-dimensional image to the display section.
 7. The image processingapparatus according to claim 1, further comprising a positioninformation acquisition section configured to acquire leading endposition information that is information indicating a leading endposition of an insertion section that is inserted into the subject,wherein the three-dimensional model structuring section and the imagegeneration section generates a three-dimensional image based on changeof the leading end position information along with an operation ofinserting the insertion section.
 8. The image processing apparatusaccording to claim 1, wherein, when generating a three-dimensional imageof the subject, the image generation section performs processing ofdifferentiating a color of an inner wall region and a color of an outerwall region in three-dimensional data structured by thethree-dimensional model structuring section.
 9. The image processingapparatus according to claim 1, wherein the image generation sectionperforms, on three-dimensional data structured by the three-dimensionalmodel structuring section, processing of smoothing the boundary regionof a lumen between an unstructured region and a structured region in athree-dimensional image of the subject and expressing the boundaryregion in a substantially curved line.
 10. The image processingapparatus according to claim 1, wherein, when generating athree-dimensional image of the subject, the image generation sectionadds index information for a surrounding region of the unstructuredregion.
 11. The image processing apparatus according to claim 1,wherein, when the unstructured region cannot be visually recognized, theimage generation section performs processing of allowing visualrecognition of the unstructured region that cannot be visuallyrecognized by performing rotation processing on three-dimensional datastructured by the three-dimensional model structuring section.
 12. Theimage processing apparatus according to claim 1, wherein, when theunstructured region cannot be visually recognized, the image generationsection performs processing of illustrating the unstructured region thatcannot be visually recognized in a color different from a color of anyother unstructured region.
 13. The image processing apparatus accordingto claim 1, wherein the image generation section performs processing ofcalculating a number of the unstructured region in three-dimensionaldata structured by the three-dimensional model structuring section, anddisplaying the number of the unstructured region on a display section.14. The image processing apparatus according to claim 1, wherein theimage generation section includes: a size calculation section configuredto calculate a size of each of the unstructured region inthree-dimensional data structured by the three-dimensional modelstructuring section; and a determination section configured to determinewhether the size calculated by the size calculation section is smallerthan a predetermined threshold, and the image generation section doesnot perform processing of allowing visual recognition on theunstructured region, the size of which is determined to be smaller thanthe predetermined threshold by the determination section.
 15. The imageprocessing apparatus according to claim 1, wherein the image generationsection performs, on three-dimensional data structured by thethree-dimensional model structuring section, processing of allowingvisual recognition of only the boundary region of a lumen between anunstructured region and a structured region in a three-dimensional imageof the subject.
 16. The image processing apparatus according to claim 1,wherein the image generation section further includes a core linegeneration section configured to generate core line data ofthree-dimensional data structured by the three-dimensional modelstructuring section, and generates, for the core line data, a core lineimage in which a region corresponding to the unstructured region has adifferent color.
 17. The image processing apparatus according to claim1, wherein the image generation section performs, on three-dimensionaldata structured by the three-dimensional model structuring section,processing of setting a color of the boundary region of a lumen betweenan unstructured region and a structured region in a three-dimensionalimage of the subject to be variable.
 18. The image processing apparatusaccording to claim 1, wherein the unstructured region is a region in thesubject that is yet to be observed with an endoscope.
 19. An imageprocessing method comprising: generating, by a three-dimensional modelstructuring section, when an image pickup signal related to a region ina subject is inputted from an image pickup apparatus configured to pickup an image of an inside of the subject, three-dimensional datarepresenting a shape of the region based on the image pickup signal; andperforming, by an image generation section, on the three-dimensionaldata generated by the three-dimensional model structuring section,processing of allowing visual recognition of a boundary region between astructured region that is a region, an image of which is picked up bythe image pickup apparatus, and an unstructured region that is a region,an image of which is yet to be picked up by the image pickup apparatus,and generating a three-dimensional image.